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To  the  Reader 


Abstract 


Who  should  use 
SAM? 


Why  was  it 
developed? 


What  is  the 
scope  of  SAM? 


How  should  it  be 
used? 


The  purpose  of  this  document  is  to  summarize  the  major  elements  of  the 
Systems  Engineering  Capability  Maturity  Model  (SE-CMM)  appraisal 
method  (SAM).  SAM  is  a  method  for  using  the  SE-CMM  to 
benchmark,  or  otherwise  appraise,  the  process  capability  of  an 
organization's  or  enterprise's  systems  engineering  function.  The  SE- 
CMM  itself  is  described  in  SECMM-94-04ICMU/SEI-94-HB-04 
[SECMM].  This  document  describes  each  step  of  an  SE-CMM 
appraisal  and  provides  guidance  for  the  preparation  and  conduct  of  an 
appraisal.  It  also  contains  background  and  context  information  about 
the  appraisal  method. 


Organizations,  enterprises,  or  projects  performing  significant  systems 
engineering  activities  are  candidates  for  using  SAM.  Organizations 
involved  in  systems  engineering  who  need  to  understand  their  use  and 
management  of  common  systems  engineering  practice  can  use  SAM  as 
the  starting  point  for  an  effort  to  improve  their  systems  engineering 
process. 


SAM  was  developed  to  provide  the  systems  engineering  community 
with  a  publicly-accessible  method  for  preparing  for  and  performing  SE- 
CMM  appraisals. 


Although  the  basic  concepts  in  SAM  are  adaptable  to  most 
organizational  appraisal  contexts,  the  scope  of  SAM  is  designed 
specifically  to  support  the  SE-CMM.  The  activity  set  for  SAM  is  the 
same  basic  set  as  that  used  by  the  SEI  CMM-Based  Appraisal  for 
Internal  Process  Improvement  (CBA-IPI)  method;  however,  differences 
from  that  method  also  exist  in  SAM  due  to  the  different  model 
representations  used  by  each  method.  This  document  is  a  process 
description  for  SAM,  not  a  training  manual.  Some  materials  are 
included  that  might  support  appraisal  training  or  the  development  of 
appraisal  training  materials;  however,  it  is  not  the  intent  of  this 
document  to  substitute  for  appraisal  training. 


The  SAM  description  is  written  to  support  facilitated  self-appraisal 
against  the  SE-CMM  as  a  reference  model.  This  SAM  description  was 
not  written  to  meet  the  needs  of  a  third-party  evaluation,  although  it 
could  be  adapted  to  that  environment  by  experienced  appraisers,  if 
necessary.  Third-party  use  of  the  method  is  not  encouraged  by  the  SE- 
CMM  Steering  Group  as  of  this  version's  release. 


continued  on  next  page 
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V 


To  th©  Roador,  continued 


Who  developed  SAM  was  developed  by  the  SE-CMM  collaboration  members,  including 
SAM?  Hughes  Aircraft  Company,  Lockheed  Corporation,  Loral  Federal 

Systems  Company,  Loral  Space  &  Range  Systems,  Software 
Engineering  Institute,  Software  Productivity  Consortium,  and  Texas 
Instruments  Incorporated,  as  part  of  their  initial  effort  to  support 
community-wide  improvement  of  systems  engineering  processes. 


Intended  The  SE-CMM  is  focused  on  four  primary  groups:  systems  engineering 

audience  practitioners  from  any  business  sector  or  government,  process  developers, 

individuals  charged  with  appraising  how  specific  systems  engineering 
organizations  implement  their  systems  engineering  processes,  and  systems 
engineering  managers.  Persons  with  five  years  or  more  of  experience  as  a 
systems  engineering  practitioner  or  manager  and  exposure  to  formal 
methods  of  organization  assessment  will  benefit  most  from  the  model  and 
appraisal  method. 


SE-CMM  The  1994  Steering  Group  for  the  SE-CMM  Project  has  provided  both 

steering  group  traditional  management  oversight  functions  and  extensive  technical  and 

members  strategic  input  to  the  project,  and  their  individual  and  collected  contributions 

to  the  project  are  appreciated  beyond  measure.  The  names  and  organization 
of  the  SE-CMM  Steering  Group  members,  as  of  May  1995,  are  provided  in 
the  table  below: 


Organization 

Contacts 

Department  of  Defense/OSD 

John  Burt 

Hughes  Aircraft  Company 

Ilene  Minnich 

Lockheed  Martin  Corporation 

Michael  Carroll 

Loral  Federal  Systems  Company 

Gary  Kennedy 

National  Institute  of  Standards  and 

Technology 

Roger  Martin 

National  Council  on  Systems  Engineering 

Don  Crocker 

Software  Engineering  Institute 

William  Peterson 

Software  Productivity  Consortium 

Art  Pyster,  PhD 

Texas  Instruments,  Incorporated 

Merle  Whatley,  PhD 

European  Software  Institute 

Colin  Tully,  PhD 

SE-CMM  Collaboration  Contacts 


continued  on  next  page 
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To  the  Reader,  continued 


Additional 
information- 
project  office 


If  you  have  any  questions  about  this  method  or  about  pilot  appraisals 
using  the  SE-CMM,  please  contact  the  SE-CMM  Project.  The 
maintenance  site  for  the  project  is  the  Software  Engineering  Institute  of 
Carnegie  Mellon  University.  The  product  managers,  Suzanne  Garcia 
and  Curt  Wells,  may  be  contacted  at: 

Suzanne  Garcia  Curtis  Wells 

Software  Engineering  Institute  Lockheed  Martin  Corporation 

4500  Fifth  Avenue  P.O.  Box  17100 

Pittsburgh,  PA  15213  Austin,  TX  78760 

(412)268-7625  (voice)  (512)386-4640  (voice) 

(412)268-5758  (fax)  (512)386-4445  (fax) 

smg@sei.cmu.edu  (email)  cwells@austin.lockheed.com  (email) 


Data  rights 
associated  with 
the  SE-CMM 


The  SE-CMM  collaboration  members  encourage  free  use  of  the  SE- 
CMM  Appraisal  Method  Description  as  a  reference  for  the  systems 
engineering  community.  Members  have  agreed  that  this  and  future 
versions  of  this  document,  when  released  to  the  public,  will  retain  the 
concept  of  free  access  via  a  permissive  copyright  notice. 
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VII 


Chapter  1:  Introduction  to  the  SE-CMM  Appraisal 
Method 


Overview  of  This  document  contains  basic  information  on  the  SE-CMM  appraisal 

document  method  (SAM).  It  is  broken  into  three  chapters  with  appendices: 

•  Chapter  1  contains  basic  context  information  and  assumptions  used  in 
creating  the  method. 

•  Chapter  2  contains  descriptions  of  each  of  the  major  process  elements 
of  SAM. 

•  Chapter  3  contains  guidance  information  that  is  helpful  in  preparing 
for  and  conducting  an  appraisal. 

•  The  appendices  contain  templates  and  instructions  for  using  support 
tools  recommended  in  SAM,  as  well  as  the  SAM  questionnaires. 


In  this  chapter  The  following  table  provides  a  guide  to  the  information  found  in  this 
chapter. 


Topic 

See  Page 

1.1  Summary  of  the  SE-CMM  Appraisal  Method 

1-2 

1.2  Assumptions 

1-9 

1.3  Appraisal  Roles 

1-10 

f 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method 


Introduction 


Appraisal 

initiation 


Typical  appraisal 
purposes 


Tailoring 
appraisal  based 
upon  objectives 


This  section  provides  a  brief  overview  of  important  aspects  of  initiating, 
preparing,  and  conducting  a  SAM.  Each  of  these  issues  is  treated  in 
more  depth  either  in  Chapter  2  or  Chapter  3. 


Appraisals  are  typically  performed  in  organizations  either  for  self 
improvement  or  supplier  selection.  Since  this  appraisal  method  is  based 
on  the  first  release  of  the  SE-CMM,  using  SAM  for  supplier  selection  is 
not  recommended. 

Process  improvement  appraisals  can  vary  significantly  depending  upon 
the  specific  objectives  of  the  appraisal.  Appraisal  objectives  will  have 
an  impact  primarily  on  the  selection  of  the  participants  and  the  duration 
of  the  appraisal  period.  Factors  to  consider  in  formulating  appraisal 
objectives  are  covered  in  Chapter  2,  Sections  2. 1.1-2. 1.3. 


The  purpose  of  appraisal  is  typically  one  of  the  following: 

•  Identify  specific  areas  for  improvement  based  upon  known  general 
areas  of  deficiency. 

•  Confirm  known  data  on  systems  engineering  practices. 

•  Obtain  buy-in  for  change  from  the  organization. 

•  Confirm  process  improvement  progress  and  determine  new  status 
(typically  second  or  subsequent  appraisals). 


The  appraisal  should  be  tailored  to  meet  the  objectives  of  the 
orgmization.  Appraisals  can  be  focused  on  almost  any  cohesive 
business  unit  of  an  enterprise:  a  specific  project,  program,  strategic 
business  unit,  functional  organization,  multiple  functional  organizations, 
or  the  entire  company.  The  selection  is  based  upon  the  sponsoring 
organization's  goals.  The  appraisal  can  also  be  focused  on  addressing 
all  of  the  process  areas,  which  is  typical  for  an  initial  appraisal,  or  it  may 
focus  on  specific  process  areas  that  may  have  been  the  target  for  process 
improvement.  Tailoring  the  number  of  process  areas  and  the  scope  of 
the  entity  being  appraised  (e.g.,  project,  functional  organizations,  etc.) 
are  the  two  aspects  that  will  have  a  significant  impact  on  the  duration 
and  resources  required  for  the  appraisal. 

Specific  aspects  of  the  appraisal  that  can  be  tailored  are  addressed  in  the 
"Tailorable  Parameters"  block  of  each  process  element  described  in 
Chapter  2. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Recording 

tailoring 

information 


Results 


In  tailoring  any  appraisal  method,  it  is  important  to  note  what  aspects 
and  parameters  of  the  method  have  been  tailored,  so  that  people  who  use 
the  data  gathered  from  that  appraisal  can  understand  the  context  from 
which  the  data  were  collected.  This  tailoring  information  should  be 
recorded  in  the  appraisal  plan. 


The  primary  work  products  of  an  SE-CMM  appraisal  are  a  findings 
briefing  and  an  appraisal  report.  The  findings  briefing  is  presented  at 
the  end  of  the  on-site  period  of  the  appraisal  and  will  include  a  process 
capability  profile  and  appraisal  findings,  which  address  both  strengths 
and  weatoesses  of  the  appraised  entity.  The  weaknesses  are  typically 
limited  to  approximately  seven  synthesized  findings  to  keep  the  scope  of 
issues  to  a  number  that  the  organization  can  manage.  The  appraisal 
report  is  written  by  the  appraisal  team  after  the  on-site  period;  it  includes 
more  detail  on  each  of  the  findings  and  specific  recommendations  for 
process  improvement  focused  on  the  findings. 

Chapter  3  discusses  the  process  capability  profile  and  other  aspects  of 
developing  appraisal  results. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Labor 

requirements 


Phases 


Table  1-1  defines  typical  labor  requirements  for  a  complete  appraisal 
(e.g.,  all  SE-CMM  process  areas  applied  to  three  to  four  projects  or 
equivalent).  Labor  requirements  can  be  tailored  as  a  function  of  the 
scope  of  the  appraisal.  A  typical  schedule  and  labor  template  is  included 
in  Appendix  D.  The  hours  for  the  facilitator  include  the  time  for  all 
phases.  The  hours  for  other  roles  are  for  the  on-site  phase  only. 
Descriptions  of  the  appraisal  roles  are  found  in  the  section  "Appraisal 
Roles"  found  later  in  this  chapter. 


Role 

Recommended 
number  of 
people 

Hours  per 
person 

Total  hours 
for  this  role 

Facilitator 

2 

60 

120 

Appraisal  team 
member  (in 
addition  to 
facilitators) 

4-6 

50 

200-300 

Systems 

engineering  leads 

1  per  project  (3 
projects) 

6 

18 

Practitioners 
from  across 
organization 

30  (3  groups  of 
10) 

4 

120 

TOTAL 

458-568 

Table  1-1.  Labor  Requirements  for  an  Appraisal. 


Table  1-2  lists  the  phases  of  the  appraisal  process.  The  process 
elements  for  each  phase  are  fully  described  in  Chapter  2. 


Phase 

Description 

Preparation 

The  activities  done  in  preparation  for  an  appraisal 

On-site 

The  activities  done  at  the  site  of  the  entity  being 
appraised 

Post  Appraisal 

The  activities  done  after  the  on-site  appraisal 
period 

Table  1-2.  Appraisal  Phases. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  continued 


Diagram  of  Figure  1-1  summarizes  the  steps  in  the  Preparation  phase. 

Preparation 

Phase 


Figure  1-1.  Diagram  of  Preparation  Phase. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  Continued 


Diagram  of  on¬ 
site  Phase 


Figure  1-2  summarizes  the  steps  in  the  On-Site  phase. 


Figure  1-2.  Diagram  of  On-Site  Phase. 


continued  on  next  page 
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1.1  Summary  of  the  SE-CMM  Appraisal  Method,  continued 


Diagram  of  Post- 
Appraisal  Phase 


Figure  1-3  summarizes  the  steps  in  the  Post- Appraisal  Phase. 


Figure  1-3.  Diagram  of  Post-Appraisal  Phase. 


continued  on  next  page 
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1.1  Summary  of  the  SE>CMM  Appraisal  Method,  Continued 


Relationship 

CBA-IPI 


to  For  those  readers  familiar  with  the  SEI  CBA-IPI  method.  Table  1-3 
shows  the  correlation  between  the  appraisal  steps  documented  in 
Chapter  2  and  the  CBA-IPI  method. 


SAM  Step 

IPI  Step 

1 . 1  Obtain  Sponsor  Commitment 

1.2  Select  Appraisal  Parameters 

1.3  Plan  Appraisal  Details 

Plan  Assessment 

1.3  Plan  Appraisal  Details 

2. 1  Conduct  Opening  Meeting 

2.2  Familiarize  Team  with  SAM 

Prepare  for  Assessment 

1.4  Collect  Data 

1.5  Analyze  Data 

2.3  Analyze  Questionnaire 

2.4  Interview  Systems  Engineering  Leads 

2.6  Interview  Practitioners 

Gather  Data 

2.5  Consolidate  Data  from  SE  Leads 

2.7  Consolidate  Data  from  Practitioners 

2.8  Develop  Preliminary  Findings 

2.9  Review  Preliminary  Findings 

2.10  Consolidate  Data 

2.13  Adjust  Draft  Findings 

Consolidate  Data 

2.5  Consolidate  Data  from  SE  Leads 

2.7  Consolidate  Data  from  Practitioners 

2.10  Consolidate  Data 

Review/Revise  Data 
Gathering  Plans 

2. 1 1  Develop  Draft  Findings 

Make  Rating  Judgments 

2. 12  Present  Draft  Findings 

2.14  Present  Final  Briefing 

2.15  Brief  Sponsor  (optional) 

2.16  Conduct  Wrap-Up 

3. 1  Report  Lessons  Learned 

3.2  Report  Appraisal  Output  to  Other  Parties 

3.3  Manage  Appraisal  Records 

Report  results 

Table  1-3.  Relationship  of  SAM  &  IPI  Activities. 
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1.2  Assumptions 


Introduction 


Assumptions 


SE-CMM  as 
reference  model 


Facilitator 

training 


SAM  focuses  on 
self-improvement 


This  section  discusses  the  underlying  assumptions,  both  procedural  and 
model-based,  that  are  relevant  to  using  SAM. 


The  SAM  description  is  based  on  the  following  assumptions: 

•  The  SE-CMM  is  the  reference  model  for  the  SAM. 

•  Readers  are  familiar  with  the  content  and  concepts  of  SECMM-94- 
04ICMU/SEI-94-HB-04,  A  Systems  Engineering  Capability  Maturity 
Model,  Version  1.0. 

•  Facilitators  intending  to  use  the  SAM  have  been  trained  in  basic 
organizational  appraisal  techniques. 

•  Although  SAM  can  be  tailored  to  either  self-improvement  or  supplier 
selection,  the  focus  and  tone  of  the  material  is  biased  toward  self- 
improvement. 


Although  the  basic  appraisal  method  described  herein  shares  many 
features  with  other  organizational  appraisal  methods,  the  data  gathering 
instruments  and  rating  development  process  are  specifically  related  to 
the  SE-CMM.  The  SE-CMM  is  fully  described  in  A  Systems 
Engineering  Capability  Maturity  Model,  Veraon  i.O  [SECMM].  This 
document  does  not  contain  specific  information  on  the  SE-CMM. 


SAM  is  an  appraisal  method  that  makes  heavy  use  of  informal 
knowledge  that  practitioners  provide  through  a  series  of  interviews  and 
feedback  sessions.  There  are  many  methods  for  interviewing  and 
synthesizing  data  from  these  types  of  data  sources.  Although  it  is  not 
necessary  for  all  members  of  the  appraisal  team  to  be  thoroughly  versed 
in  the  concepts  of  organizational  appraisal,  the  appraisal  will  proceed 
more  smoothly,  and  is  likely  to  produce  better  results,  if  at  least  one, 
and  preferably  two,  of  the  appraisal  team  members  are  skilled  in  the 
facilitator  role  for  organizational  appraisals.  The  facilitators  are  included 
as  members  of  the  appraisal  team. 


The  actual  character  of  each  appraisal  will  differ  based  on  differences  in 
organizations'  cultures  and  other  business  contexts.  The  process 
described  focuses  on  a  feedback  loop  that  includes  multiple  levels  of  the 
entity  being  appraised.  This  is  one  of  the  ways  in  which  the  SAM 
contributes  to  an  overall  process  improvement  focus  in  the  organization. 
There  are  many  other  models  for  organizational  improvement,  and  most 
include  a  benchmarking  step  that  serves  to  determine  the  current  state  of 
the  organization.  Many,  like  SAM,  use  this  benchmarking  step  as  a 
way  to  focus  attention  and  resources  on  the  improvement  effort. 
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1.3  Appraisal  Roles 


Personnel  and  Table  1-4  defines  the  personnel  and  associated  roles  that  are  typically 
associated  roles  involved  in  an  appraisal.  The  responsibilities  and  functions  performed 
by  each  of  the  roles  is  further  defined  in  Chapter  2.  Note  that  multiple 
roles  may  be  assigned  to  one  person  as  appropriate. 


Role  Name 

Description 

Appraisal 

team 

The  appraisal  team  consists  of  those  who  conduct  the 
appraisal.  All  of  the  personnel  on  the  appraisal  team 
should  be  familiar  with  the  SE-CMM  prior  to  the  on¬ 
site  phase.  This  group  includes  the  following  roles: 

•  Facilitator. 

•  Appraisal  team  leader. 

•  Site  coordinator. 

•  Appraisers  drawn  from  the  entity  being  appraised 
(internal)  or  customer  (supplier  selection). 

The  term  appraisal  team  members  indicates  any  of  the 
roles  cited  above 

Facilitator 

Member  of  the  appraisal  team  who  is  responsible  for 

•  Famiharizing  the  appraisal  team  with  SAM. 

•  Facilitating  the  appraisal  process  during  the  on-site 
phase. 

•  Providing  SE-CMM  expertise. 

The  facilitator  is  often  drawn  from  outside  the 
sponsoring  organization. 

Appraisal 
team  leader 

Individual  who  is  responsible  for 

•  Presenting  the  appraisal  findings  and  developing  the 
appraisal  report. 

Site 

coordinator 

Individual  who  is  responsible  for 

•  Obtaining  facilities  for  the  on-site  phase. 

•  Scheduling  activities  during  the  on-site  week. 

•  Administering  and  collecting  questionnaires. 

•  Ensuring  personnel  attendance,  as  appropriate. 

Appraiser 

An  appraisal  team  member  who  participates  in  the  data 
collection,  analysis,  findings  generation,  and 
communication  with  the  participants  from  the 
organization  (appraised  entity). 

Table  1-4.  Appraisal  Roles. 


continued  on  next  page 
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1 .3  Appraisal  Roles,  Continued 


Personnel  and 
associated  roles 
(continued) 


Role  Name 

Description 

Appraisal 

participants 

The  appraisal  participants  are  the  subjects  of  data 
gathering  in  an  appraisal.  This  group  includes 

•  Project  lead  systems  engineers  (or  equivalent). 

•  Practitioners  (both  technical  and  management). 

•  Support  personnel. 

Participants  are  the  primary  sources  of  data  via 
questionnaires  and  interviews.  Guidance  on  participant 
selection  is  found  in  Chapter  2. 

Project  lead 
systems 
engineers  (or 
equivalent) 

Appraisal  participants  identified  as  having 
responsibility  for  the  systems  engineering  aspects  of  a 
project.  The  SE  lead  should  have  broad  knowledge  of 
the  full  life  cycle  of  product  development.  The  systems 
engineering  leads 

•  Complete  the  SAM  questionnaire. 

•  Participate  in  a  series  of  question  and  answer 
sessions. 

•  Are  a  primary  source  of  feedback  on  the  validity  of 
the  findings. 

Practitioners 

Individuals  who  perform  or  support  the  systems 
engineering  process  (direct  and  indirect,  e.g.,  training, 
customers,  and  suppliers).  Practitioners  are  a  source 
of  data,  primarily  via  the  interview  and  also  as  a 
reviewer  of  the  appraisal  findings.  They  could  also 
include  questionnaire  respondents. 

Appraisal 

support 

personnel 

Personnel  who  support  the  appraisal  process. 

Examples  include  secretaries  and  logistics  coordinators 
for  the  appraisal. 

Appraisal 

customer 

Individual  or  group  who  defines  the  objectives, 
receives  the  results,  and  covers  the  cost  of  the 
appraisal.  This  group  includes 

•  Sponsor 

•  Management 

Table  1-4.  Appraisal  Roles,  continued 
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1.3  Appraisal  Roles,  Continued 


Personnel  and 
associated  roles 
(continued) 


Role  Name 

Description 

Sponsor 

Individual  providing  the  resources  for  the  appraisal  and 
the  commitment  to  the  process  improvement  effort,  in 
the  case  of  self-improvement.  It  is  particularly 
important  for  the  sponsor  to  show  commitment  by 
attending  both  the  opening  and  closing  meetings  of  the 
on-site  phase. 

Management 

Management  of  the  appraised  entity,  both  at  the 
organization-  and  project-level.  They  are  recipients  of 
the  appraisal  findings  and  the  primary  role  responsible 
for  carrying  out  improvements  suggested  by  the 
appraisal. 

Appraisal 

champion 

Individual  who  initiates  the  dialogue  within  the  sponsor 
and  management  groups  and  obtains  the  initial 
sponsorship  for  the  appraisal.  The  champion  usually 
plays  a  role  in  interfacing  the  site  coordinator  and  the 
sponsor  in  removing  any  obstacles  to  the  appraisal  that 
may  be  encountered. 

Table  1-4.  Appraisal  Roles,  continued 
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Chapter  2:  Summaries  of  SAM  Process  Elements 


Introduction 


In  this  section 


This  chapter  contains  summaries  of  the  major  phases  and  process 
elements  of  the  SAM. 


The  following  process  elements  are  addressed  in  this  chapter: 


ID 

Process  Element  Name 

See  Page 

2.1 

Preparation 

2-3 

2.1.1 

Obtain  Sponsor  Commitment 

2-6 

2.1.2 

Select  Appraisal  Parameters 

2-8 

2.1.3 

Plan  Appraisal  Details 

2-10 

2.1.4 

Collect  Questionnaire  Data 

2-12 

2.2 

On  Site 

2-14 

2.2.1 

Conduct  Opening  Meeting 

2-20 

2.2.2 

Familiarize  Team  with  SAM 

2-22 

2.2.3 

Analyze  Questionnaire 

2-24 

2.2.4 

Interview  Systems  Engineering  Leads 

2-27 

2.2.5 

Consolidate  Data  from  SE  Leads 

2-30 

2.2.6 

Interview  Practitioners 

2-32 

2.2.7 

Consolidate  Data  from  Practitioners 

2-37 

2.2.8 

Develop  Preliminary  Findings 

2-40 

2.2.9 

Review  Preliminary  Findings 

2-42 

2.2.10 

Develop  Draft  Rating 

2-45 

2.2.11 

Develop  Draft  Findings 

2-48 

2.2.12 

Present  Draft  Findings 

2-51 

2.2.13 

Adjust  Draft  Findings 

2-55 

2.2.14 

Present  Final  Briefing 

2-58 

2.2.15 

Brief  Sponsor  (optional) 

2-60 

2.2.16 

Conduct  Wrap-Up 

2-62 

2.3 

Post-Appraisal 

2-64 

2.3.1 

Report  Lessons  Learned 

2-66 

2.3.2 

Report  Appraisal  Output  to  Other  Parties 

2-67 

2.3.3 

Manage  Records 

2-68 
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Chapter  2: 

Continued 


How  to  read  the 
process  element 
summaries 


Summaries  of  SAM  Process  Elements, 


All  process  element  descriptions  are  similarly  formatted,  and  contain 
several  blocks,  which  are  described  in  Table  2-1  below.  For  each  major 
phase  (preparation,  on-site,  post-appraisal),  the  summary  description 
contains  the  process  elements  that  comprise  that  phase;  for  the  other 
process  elements,  the  process  element  summary  contains  the  actual 
description  of  that  element. 


Block  Title 

Description 

Element  title/tag 

The  SAM  title  for  the  process  element,  the  code 
for  the  process  element,  and  the  number  of  the 
element  used  for  reference 

Purpose 

The  major  purpose  for  this  process  element  within 
SAM 

Major  activities 

An  overall  summary  of  the  activities  associated 
with  the  process  element 

Major  participants 

The  major  roles  involved  in  the  process  element 
and  a  summary  of  their  responsibilities  associated 
with  the  process  element 

Typical  duration 

A  range  of  the  typical  time  duration  (e.g.,  number 
of  hours)  expected  for  the  process  element 
described 

Steps 

The  substeps  for  the  element  along  with  guidance 
for  their  performance,  if  appropriate 

Tailorable 

parameters 

The  parameters  associated  with  this  process 
element  that  are  expected  to  be  tailored  for 
different  appraisal  goals 

Exit  criteria 

A  description  of  the  decision-making  criteria  to 
determine  if  the  process  element  has  been 
completed 

Notes 

Notes  on  the  process  element  that  do  not  fit  in  any 
of  the  other  categories 

Table  2-1.  How  to  Read  Process  Element  Descriptions. 
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2.1  Preparation 


Element 

tag 

Purpose 


Diagram 


title/  2. 1  Preparation  (PR) 


The  purpose  of  the  Preparation  phase  is  to  prepare  the  sponsor,  the 
apprais^  team,  and  the  appraised  entity  for  the  on-site  period.  Major 
elements  of  preparation  include 

•  The  sponsor  must  commit  to  the  resource  requirements  and  action 
requirements  for  the  appraisal  process. 

•  The  team  must  be  selected  to  perform  the  appraisal. 

•  The  appraised  entity  must  provide  preliminary  data  to  the  team  for 
analysis. 

•  The  team  must  analyze  the  data  prior  to  arriving  on-site. 

•  Figure  2-1  shows  the  major  steps  in  the  preparation  phase. 


Figure  2-1.  Diagram  of  Preparation  Phase. 


continued  on  next  page 
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2.1  Preparation,  Continued 


Summary 

description 


Table  2-2  lists  the  major  activities  of  the  Preparation  phase  and  the 
expected  output  of  each.  Each  element  is  described  more  fully  in  the 
summaries  that  follow. 


ID 

Activity 

Description 

Output 

2.1.1 

Obtain 

sponsor 

commitment 

Get  the  sponsor  to 

•  agree  to  the  concept 
of  the  appraisal 

•  define  goals  of  the 
appraisal  in  relation  to 
the  sponsor's 
business  goals 

•  agree  to  provide 
necessary  resources, 
including  funding 

•  understand  his/her 
role  in  sponsoring  the 
improvement  activity 
implied  by  engaging 
in  the  appraisd 

•  Appraisal 
resources 

•  Agreement  to 
proceed  and 
expend  resources 
for  the  planning 
and  conduct  of 
the  appraisal 

•  Appraisal  goals 
related  to 
business  goals 

2.1.2 

Select 

appraisal 

parameters 

Select  appraised  entity 
and  appraisal  team,  and 
tailor  appraisal  method 
as  required  (e.g.,  team 
size,  site  time,  ratings 
to  produce,  number  of 
projects,  project 
characteristics) 

•  Preliminary 
appraisal  plan  that 
defines  appraisal 
parameters  based 
on  appraisal  goals 

2.1.3 

Plan  appraisal 
details 

Use  appraisal 
parameters  to  develop 
schedule,  training 
requirements,  and 
appraisal  support 
materials;  to  select 
training  instruments;  to 
develop  and  train 
appraisal  team 

•  Approved 
appraisal  plan 

2.1.4 

Collect  data 

Administer 
questionnaire  and 
review  any  appropriate 
documents/artifacts 

•  Responses  to 
questionnaire 

•  Document/artifact 
review  notes 

Table  2-2.  Summary  Description  of  Preparation  Phase. 


continued  on  next  page 
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2.1  Preparation,  continued 


Major 

participants 


Typical  duration 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-3  lists  the  primary  roles  involved  in  the  preparation  phase  and  a 
summary  of  their  activity  during  this  phase. 


Role 

Summary 

Sponsor 

Makes  the  decision  to  commit  to  appraisal  and 
provides  resource  commitments  and  tailoring 
guidelines  for  the  appraisal. 

Facilitator/site 

coordinator 

Interacts  with  the  management  and  personnel 
at  the  site  to  obtain  commitment  for  the 
appraisal  and  ensure  that  the  planning  for  the 
appraisal  is  successfully  completed. 

Appraisal  team 

Interacts  with  management  and  the  appraisal 
leader  to  perform  the  planning  and  other 
preparation  activities  for  the  appraisal. 

Table  2-3.  Participants  for  Preparation. 


Four  to  eight  weeks,  depending  on  the  complexity  of  the  appraisal 
selected  and  other  site  parameters 


•  Number  of  site  visits  prior  to  on-site  period. 

•  Number  of  meetings  with  senior  management. 

•  Number  of  meetings  with  appraisal  team. 

•  Appraisal  goals. 

•  Number  of  projects  to  appraise. 

•  Number  of  appraisal  team  members. 

•  Process  area/capability  level  focus  for  the  appraisal. 


•  Appraisal  goals  established. 

•  Appraisal  resources  committed. 

•  Project,  team,  participant  selection  completed. 

•  Appraisal  plan  approved. 

•  Preliminary  data  gathering  completed. 


This  is  where  the  overall  context  for  the  appraisal  is  established.  The 
decisions  made  here  will  have  an  impact  on  the  rest  of  the  appraisal 
activities.  All  relevant  decisions  from  this  phase  should  be  documented 
in  the  appraisal  plan. 
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2.1.1  Obtain  Sponsor  Commitment 


Element  title/ 
tag 


Purpose 


Summary 

description 


Major 

participants 


2.1.1  Obtain  Sponsor  Commitment  (OC) 


The  purposes  of  Obtain  Sponsor  Commitment  are  to  establish  the 
sponsor's  commitment  to  the  appraisal  and  to  determine  the  major  goals 
of  the  appraisal,  which  will  guide  the  selection  of  appraisal  parameters. 


Obtain  Sponsor  Commitment  involves  meeting  with  the  sponsors  of  the 
appraisal  to  provide  an  understanding  of  the  concepts  of  the  SE-CMM 
and  the  SE-CMM  appraisal  method  (SAM),  engaging  the  sponsor  in 
dialogue  to  determine  his/her  goals  for  the  appraisal,  and  negotiating  a 
commitment  for  resources,  including  personnel  and  funding  for  the 
appraisal  activities.  A  primary  aspect  of  obtaining  sponsor  commitment 
is  ensuring  the  sponsor  understands  his/her  role  in  the  appraisal. 
Chapter  3  contains  additional  guidance  on  establishing  appraisal  goals. 


Table  2-4  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Sponsor 

Engages  in  dialogue  with  appraisal 
leader/champion  to  understand  appraisal 
context  and  set  appraisal  goals;  conomits 
resources  for  the  appraisal. 

Appraisal  champion 

Establishes  link  between  sponsor,  appraisal 
team  leader,  and  potential  appraisal 
participants. 

Appraisal  team 
leader 

Provides  sponsor  with  information  needed  to 
make  a  commitment  decision  on  the  appraisal; 
provides  a  link  between  the  appraisal 
champion  and  potential  appraisal  participants. 

Table  2-4.  Participants  for  Obtain  Sponsor  Commitment. 
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2.1.1  Obtain  Sponsor  Commitment,  continued 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


Two  hours  to  several  weeks 


•  Method  of  interaction  with  sponsor;  all  face-to-face  meetings, 
combination  of  telephone/video  and  face  to  face  meetings, 
combination  of  written  communication  and  face-to-face  meetings,  etc. 

•  Number  of  interactions  with  sponsor. 


•  Appraisal  goals  established. 

•  Sponsor  commitment  to  provide  appraisal  resources  obtained. 

•  Sponsor  commitment  to  appropriate  behavior  during  the  appraisal 
obtained. 


This  is  a  go/no-go  decision  point.  If  sponsor  commitment  for  the 
appraisal  is  not  obtained,  no  further  process  elements  related  to  SAM 
will  be  performed. 

Guidance  on  using  business  goals  and  organization  context  for  tailoring 
the  appraisal  is  addressed  in  Chapter  3. 
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2.1.2  Select  Appraisal  Parameters 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 


2.1.2  Select  Appraisal  Parameters  (SP) 


The  purpose  of  Select  Appraisal  Parameters  is  to  determine  how  the 
SAM  needs  to  be  tailored  in  order  to  meet  the  goals  established  for  the 
appraisal  with  the  sponsor. 


Select  Appraisal  Parameters  involves  determining  the  membership  of  the 
appraisal  team,  the  profile  of  projects  to  be  selected  from  the  appraised 
entity,  and  the  major  appraisal  participants.  In  addition,  a  preliminary 
appraisal  plan  is  produced  that  documents  the  tailoring  of  SAM  process 
elements. 


Table  2-5  lists  the  primary  roles  involved  in  this  process  element  and  the 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team  leader 
(ATL)  or  facilitator 

Engages  in  dialogue  with  the  appraised 
entity's  management  to  determine  the 
project  profile  for  the  appraisal;  engages 
with  the  appraising  organization  to 
determine  the  makeup  of  the  appraisal  team 
(in  the  case  of  self-improvement,  this  is  the 
same  as  the  appraised  entity).  Produces 
plan. 

Sponsor 

Engages  in  dialogue  with  ATL  and 
provides  appropriate  input  on  team  and 
appraised  entity  characteristics. 

Table  2-5.  Participants  for  Select  Appraisal  Parameters. 


One  to  two  weeks,  depending  on  complexity  of  tailoring  required 


•  Depth  of  documentation  of  tailoring  decisions. 
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2.1.2  Select  Appraisal  Parameters,  continued 


Exit  criteria 


Notes 


•  Appraisal  team  leader  selected. 

•  Appraisal  team  selected. 

•  Projects  to  be  appraised  selected. 

•  Appraisal  participants  selected. 

•  Preliminary  appraisal  plan  developed. 


This  is  the  step  where  tailoring  decisions  are  made  and  documented 
based  on  the  goals  of  the  appraisal. 

Guidance  on  tailoring  the  appraisal  to  support  the  organization's 
business  goals  and  organizational  context  is  address  in  Chapter  3. 
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2.1.3  Plan  Appraisal  Details 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 


2.1.3  Plan  Appraisal  Details  (PD) 


The  purpose  of  Plan  Appraisal  Details  is  to  produce  and  obtain  approval 
for  the  final  appraisal  plan,  which  documents  the  parameters  and  details 
of  the  appraisal. 


Plan  Appraisal  Details  involves  establishing  the  availability  of  planned 
interviewees  during  the  on-site  period,  planning  the  logistics  of  the 
appraisal  (meeting  rooms,  support  staff  availability,  etc.),  and  verifying 
the  schedule  for  the  appraisal  with  all  affected  parties.  It  also  involves 
verifying  who  will  receive  data  on  the  conclusions  of  the  appraisal  and 
establishing  feedback  mechanisms  for  lessons  learned. 


Table  2-6  lists  the  primary  roles  involved  in  this  process  element  and  the 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team  leader 

Oversees  production  of  and  approval  for 
final  appraisal  plan. 

Appraisal  team 

Produces  assigned  sections  of  appraisal 
plan. 

Sponsor 

Approves  final  appraisal  plan. 

Appraisal  champion 

Stays  in  touch  with  site  coordinator  on 
progress  of  preparation  and  assists  in 
removing  obstacles  for  the  appraisal. 

Site  coordinator 

Verifies  the  schedules  of  the  intended 
participants,  arranges  logistics  details  for 
the  appraisal. 

Table  2-6.  Participants  for  Plan  Appraisal  Details. 


Two  to  six  weeks,  depending  on  the  complexity  of  the  appraisal  plan 


•  Degree  to  which  details  are  planned  (depends  on  the  complexity  of  the 
appraisal). 
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2.1 .3  Plan  Appraisal  Details,  Continued 


Exit  criteria 


Notes 


•  Appraisal  schedule  established. 

•  Appraisal  participant  availability  confirmed. 

•  Appraisal  plan  approved. 

•  Logistics  of  appraisal  prepared. 


This  is  when  the  actual  schedule  for  the  on-site  phase  is  produced.  (See 
Appendix  F,  Site  Coordinator  Checklist,  for  details  on  preparing  for  the 
logistics  of  an  appraisal.) 

If  the  appraisal  champion  has  not  become  an  appraisal  team  member, 
he/she  is  likely  to  keep  in  touch  with  the  appraisal  team  leader 
throughout  the  appraisal  preparation  phase. 

Any  useful  documents,  such  as  policies,  process  descriptions  (often  in 
training  materials),  standards,  even  meeting  minutes  or  action  item  lists, 
may  be  requested,  collected,  and  stored  for  use  by  the  appraisal  team. 
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2.1.4  Collect  Questionnaire  Data 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


2.1.4  Collect  Questionnaire  Data  (CQ) 


The  purpose  of  Collect  Questionnaire  Data  is  to  obtain  profile 
information  on  the  appraised  entity  and  to  administer  and  collect  data 
from  the  questionnaire.  Preliminary  transference  of  the  questionnaire  to 
the  DTS  can  also  be  performed  as  part  of  this  step  if  the  necessary 
resources  are  available. 


Collect  Questionnaire  Data  involves  administering  and  collecting  results 
from  the  questionnaire  and  instruments  used  to  profile  the  organization. 
The  questionnaire  rephrases  the  base  and  generic  practices  of  the  SE- 
CMM  into  a  form  that  is  appropriate  for  data  gathering.  Its  results  are 
used  to  focus  the  on-site  data  gathering  for  the  appraisal.  The 
questionnaire  responses  are  transferred  to  the  data  tracking  sheet  either 
in  this  step  or  in  the  "Analyze  Questionnaire"  step  in  Chapter  2. 


Table  2-7  lists  the  primary  roles  involved  in  this  process  element  and  the 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  participants 

The  selected  participants  provide  data  on 
projects  and  the  organization  via  profile 
questions  and  the  questionnaire.  The 
questionnaire  may  be  administered  in 
various  ways,  e.g.,  all  participants 
together  in  a  face-to-face  meeting,  as  an 
on-line  response  activity,  or  mail-ins.  It  is 
recommended  that  the  questionnaire  be 
administered  as  a  group,  with  the  facilitator 
or  other  person  with  knowledge  about  the 
SE-CMM  present  to  answer  questions  of 
terminology,  etc. 

Appraisal  team 

Determines  recipients  of  the  questionnaires 
and  oversees  administration  of  them. 

Appraisal  team  leader 

Conducts  any  meetings  related  to 
introducing  the  data  collection  activity. 

Sponsor 

Ensures  that  all  participants  are  made 
available  for  data  collection. 

Table  2-7.  Participants  for  Collect  Questionnaire  Data. 
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2.1.4  Collect  Questionnaire  Data,  Continued 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


One  to  two  hours  per  participant 


•  Method  of  administering  questionnaires. 

•  Contents  of  questionnaire,  based  on  appraisal  goals  (e.g.,  if  only 
appraising  process  areas  1,  3,  7,  only  use  those  questionnaire  items). 

•  Amount  of  analysis  performed  prior  to  on-site  phase. 

•  Transference  of  questionnaire  responses  to  the  DTS. 


•  Project  profile  information  received  by  appraisal  team. 

•  Filled-in  questionnaires  received  by  appraisal  team. 

•  Optional:  if  it  is  decided  to  transfer  questionnaire  responses  prior  to 
the  on-site  phase,  the  transference  should  be  completed  at  the  end  of 
this  process  element. 


See  the  questionnaire  distribution  table  in  Section  3.3  and  the  SAM 
questionnaire  instructions  in  the  appendix  for  information  on  using  the 
questionnaire  to  collect  data.  See  Section  2.2.3  for  details  on 
transferring  the  questionnaire  responses  to  the  DTS. 
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2.2  On  Site 


Element  title/tag 
Purpose 


2.2  On  Site 


The  purpose  of  the  on-site  phase  is  to  explore  the  results  of  the 
preliminary  data  analysis,  provide  an  opportunity  for  practitioners  at  the 
appraised  entity  to  participate  in  the  data  gathering  and  validation 
process,  and  provide  practitioners  and  management  with  input  on  the 
results  of  the  appraisal.  Figure  2-2  shows  the  steps  in  the  On-Site 
phase. 


continued  on  next  page 


2-14 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


2.2  On  Site,  Continued 


Diagram 


Figure  2-2.  Diagram  of  On-Site  Phase. 
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2.2  On  Site,  Continued 


Summary 

description 


Table  2-8  lists  the  major  activities  of  the  on-site  phase  and  the  expected 
output  of  each.  Each  element  is  described  more  fully  in  the  summaries 
that  follow. 


ID 

Activity 

Description 

Outputs 

2.2.1 

Conduct 

opening 

meeting 

The  opening  meeting 
provides  participants  with 
an  overview  of  the 
appraisal  and  reminds 
them  of  the  context  of  the 
appraisal  (e.g.,  schedule 
and  confidentiality).  It  is 
also  the  time  where  the 
sponsor  expresses  support 
of  the  appraisal  and 
subsequent  process 
improvement  activities. 

•  Supported 
appraisal  effort 

•  Answered 
questions 

2.2.2 

Familiarize 
team  with 
SAM 

The  facilitator  instructs  the 
appraisal  team  on  the 
detailed  conduct  of  the 
appraisal  activities  and 
introduces  the  use  of  the 
model  for  appraisal. 

•  Prepared  team 

2.2.3 

Analyze 

question¬ 

naire 

The  appraisal  team 
analyzes  the  responses  to 
the  questionnaire  and 
formulates  a  set  of  follow- 
on,  exploratory  questions 
for  the  systems 
engineering  leads. 

•  Exploratory 
questions 

2.2.4 

Interview 

systems 

engineering 

leads 

Through  structured 
interview  techniques  using 
the  exploratory  questions, 
the  appraisal  team  gathers 
corroborating  data 
regarding  the  project's 
systems  engineering 
practices. 

•  Interview  notes 

•  Data  requests 

2.2.5 

Consolidate 
data  from  SE 
leads' 
interviews 

The  team  members  review 
their  notes,  discuss  any 
issues,  and  update  their 
data  tracking  sheets 
(DTSs). 

•  Updated  DTS 

•  Adjustments  to 
practitioner 
interviews 

Table  2-8.  Summary  Description  of  the  On-Site  Phase. 
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2.2  On  Site,  Continued 


Summary 

description, 

continued 


ID 

Activity 

Description 

Outputs 

2.2.6 

Interview 

practitioners 

Through  open-ended, 
facilitated  discussion,  the 
team  gathers  corroborating 
data  on  organizational 
practices  from  different 
types  of  practitioners. 

•  Interview  notes 

2.2.7 

Consolidate 
data  from 
practitioner 
interviews 

The  team  members  review 
their  notes,  discuss  any 
issues,  and  update  their 
data  tracking  sheets. 

•  Updated  DTS 

2.2.8 

Develop 

preliminary 

findings 

Using  all  data  sources 
avziilable,  the  team 
generates  a  preliminary 
list  of  findings  with  regard 
to  the  organization's 
systems  engineering 
practices. 

•  Preliminary 
findings 

2.2.9 

Review 

preliminary 

findings 

The  team  provides 
preliminary  findings  to 
systems  engineering  leads 
to  validate  that  what  they 
heard  is  correct. 

•  Notes  from 
presentations 

•  Validated/cor¬ 
rected  findings 

2.2.10 

Develop  draft 
rating 

The  team  members  review 
their  notes,  discuss  any 
issues,  update  their  data 
tracking  sheets,  and 
formulate  draft  ratings. 

•  Updated  DTS 

•  Draft  ratings 

2.2.11 

Develop  draft 
findings 

The  team  prioritizes  and 
provides  wording  for 
findings  that  fit  the 
appraisal  context. 

•  Prioritized  draft 
findings 

Table  2-8.  Summary  Description  of  the  On-Site  Phase,  continued 
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2.2  On  Site,  Continued 


Summary 

description, 

continued 


ID 

Activity 

Description 

Outputs 

2.2.12 

Present  draft 
findings 

The  team  reports  appraisal 
findings  to  the  appraisal 
practitioners  and  systems 
engineering  leads. 

•  Practitioner 
feedback 

•  Feedback  from 
systems 

engineering  leads 

•  Proposed 
adjustments  to 
findings 

2.2.13 

Adjust  draft 
findings 

Based  on  participants’ 
feedback,  the  findings  are 
adjusted  for  final 
presentation.  At  this  time, 
the  process  area  (PA) 
ratings  are  reviewed  for 
appraisal  team  consensus. 

•  Final  briefing 

•  Findings 

•  PA  ratings 

2.2.14 

Present  final 
briefing 

The  final  findings  and 
process  capability  profile 
are  presented  to  all 
participants  in  the  presence 
of  the  sponsor.  Future 
activities  are  discussed. 

•  Participant  buy-in 

•  Participant 
expectations 

2.2.15 

Brief 

sponsor 

(optional) 

The  team  provides  the 
sponsor  with  the 
opportunity  to  discuss  the 
results  with  the  appraisal 
team  privately. 

•  Sponsor  buy-in 

•  Sponsor 
expectations 

2.2.16 

Conduct 

wrap-up 

The  appraisal  team 
discusses  post  on-site 
activities.  They  also 
provide  feedback 
regarding  the  SE-CMM 
and  SE-CMM  appraisal 
method  for  use  by  the  SE- 
CMM  collaboration. 

•  Follow-on  plans 

•  SE-CMM 
feedback 

Table  2-8.  Summary  Description  of  the  On-Site  Phase,  continued 
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2.2  On  Site,  Continued 


Typical  duration  Five  calendar  days 


Tailorable  •  Number  of  interviews, 

parameters  •  Number  and  type  of  document  reviews. 

•  Amount  of  direction  in  interviews. 


Exit  criteria  •  Interviews  completed. 

•  Document  reviews  completed. 

•  Findings  briefing  delivered. 

•  Appraisal  report  planned. 

•  Feedback  on  SE-CMM  and  SAM  collected. 


Notes  The  following  diagram  provides  an  overview  of  a  typical  five-day  on¬ 

site  appraisal  period.  This  example  assumes  four  projects  and  three 
practitioner  groups. 


Figure  2-3.  Typical  On-Site  Appraisal  Schedule. 
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2.2.1  Conduct  Opening  Meeting 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2. 1  Conduct  Opening  Meeting  (OM) 


The  purpose  of  Conduct  Opening  Meeting  is  to  present  the  appraisal 
process  and  schedule  to  the  sponsor  and  all  appraisal  participants.  An 
additional  purpose  is  for  the  sponsor  to  express  support  for  the  appraisal 
activities. 


Conduct  Opening  Meeting  involves  gathering  all  the  appraisal 
participants  together,  along  with  the  sponsor  or  customer  of  the 
appraisal  (depending  on  appraisal  context),  to  review  the  appraisal 
process  and  reaffirm  the  sponsor's  commitment  to  the  appraisal. 


Table  2-9  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team  leader 

Welcomes  participants  and  introduces 
sponsor. 

Sponsor 

Shows  management  support  for  the 
appraisal  and  subsequent  process 
improvement  activities. 

Facilitator 

Presents  brief  overview  of  the  model  and 
appraisal. 

Site  coordinator 

Presents  schedules  and  locations. 

Appraisal  team 

Supports  appraisal  team  leader. 

Appraisal  participants 

Learn  their  role  in  the  on-site  phase  of 
appraisal  activities. 

Table  2-9.  Participants  for  Conduct  Opening  Meeting. 


1.5  hours 
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2.2.1  Conduct  Opening  Meeting,  continued 


Steps 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-10  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Sponsor  comments 

The  sponsor  expresses  support  for  the 
appraisal  and  commitment  to  the  resulting 
process  improvement  recommendations. 

SE-CMM  introduction 

The  facilitator  gives  a  brief  introduction  to 
the  SE-CMM. 

Appraisal  process 

The  facilitator  presents  an  overview  of  the 
appraisal  process.  Confidentiality  rules  are 
explained. 

Schedule  review 

The  site  coordinator  reviews  the  schedule 
and  locations  for  the  week's  activities  and 
stresses  the  necessity  of  being  on  time. 

Question  &  answer 

The  sponsor,  facilitator,  and  appraisal  team 
leader  answer  any  questions  from  the 
appraisal  participants. 

Table  2-10.  Steps  for  Conduct  Opening  Meeting. 


•  The  presentation  will  vary  based  on  the  purpose  of  the  appraisal.  In 
any  case,  the  way  in  which  the  results  will  be  used  should  be  a  part  of 
this  presentation. 


•  Opening  briefing  delivered. 

•  Questions  of  appraisal  participants  answered. 


Refer  to  Appendix  A  for  a  sample  opening  briefing. 

The  appraisal  goals  and  use  of  results  is  established  in  the  preparation 
phase  as  part  of  Select  Appraisal  Parameters. 
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2.2.2  Familiarize  Team  with  SAM 


Element  title/tag 
Purposes 

Summary 

description 

Major 

participants 


Typical  duration 


2.2.2  Familiarize  Team  with  SAM  (FT) 


The  purpose  of  Familiarize  Team  with  SAM  is  to  prepare  the  appraisal 
team  for  performing  its  role  in  the  appraisal  process. 


This  is  an  opportunity  for  the  appraisal  team  to  begin  to  work  together. 
The  briefing  begins  with  a  review  of  the  SE-CMM.  The  appraisS  steps 
are  presented  in  greater  detail  than  at  the  opening  meeting,  and  the 
team's  role  in  each  step  is  clarified. 


Table  2-11  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Presents  the  familiarization  briefing  and 
answers  any  questions. 

Appraisal  team 

Brings  up  any  questions  on  the  model  and 
appraisal  method. 

Table  2-11.  Participants  for  Familiarize  Team  with  SAM. 


Two  hours 
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2.2.2  Familiarize  Team  with  SAM,  continued 


Steps 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-12  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Team  building 

The  facilitator  leads  the  team  in  a  team-building 
exercise;  at  the  very  least,  all  team  members 
introduce  themselves. 

Review  SE-CMM 

The  facilitator  presents  the  salient  features  of 
the  model  and  its  use  in  appraisal  and  discusses 
them  with  the  team. 

Review  SAM 

The  facilitator  explains  each  step  in  the 
appraisal  process.  The  team's  role  is  described 
with  particular  emphasis  on  behavior  and  note¬ 
taking  procedures  to  be  followed  during 
interviews.  The  facilitator  presents  the 
techniques  that  will  be  followed  to  manage 
data,  develop  results,  and  reach  consensus. 

Question  and  answer 

The  facilitator  responds  to  the  team's  questions 
concerning  the  model  and  the  appraisal  process. 

Table  2-12.  Steps  for  Familiarize  Team  with  SAM. 


•  Depth  of  instruction  on  model  and  SAM,  depending  on  experience  of 
team. 

•  Types  of  tools  to  use  to  support  the  appraisal. 


•  Appraisal  team  understands  use  of  SE-CMM  in  appraisal. 

•  Appraisal  team  understands  its  role  in  appraisal  activities. 

•  Appraisal  team  commits  to  performing  the  appraisal  as  structured  in 
preparation  phase. 


Because  there  is  no  formal  training  available  for  v  1.0  of  SAM,  this  step 
is  important  to  ensure  that  the  appraisal  team  understands  the  basic  flow 
of  the  appraisal  activities  and  their  responsibilities  throughout  the  week. 
See  Appendix  E  for  some  support  materials  useful  in  familiarizing  the 
appraisal  team  with  SAM.  Other  appendices  contain  support  materials 
(e.g.,  data  tracking  sheet)  that  the  team  may  wish  to  use. 
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2.2.3  Analyze  Questionnaire 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.3  Analyze  Questionnaire  (AQ) 


The  purpose  of  Analyze  Questionnaire  is  to  develop  a  set  of  exploratory 
questions  for  use  in  the  interviews  with  the  systems  engineering  leads. 


The  appraisal  team  analyzes  the  responses  to  the  questionnaire  to 
determine  areas  for  practice  validation  and  potential  discrepancy,  and  to 
perform  a  gap  analysis  against  the  SE-CMM.  If  not  completed 
previously,  the  questionnaire  responses  are  transcribed  to  the  data 
tracking  sheets.  This  process  element  results  in  a  set  of  candidate 
exploratory  questions  (EQ)  for  the  interview  with  the  systems 
engineering  leads,  and  "listen  fors"  at  practitioner  sessions.  Training 
materials,  or  project-specific  data  such  as  plans,  meeting  agendas,  or 
action  item  lists,  that  have  been  collected  may  be  reviewed  and  used  to 
support  EQ  development  or  corroborate  questionnaire  answers. 


Table  2-13  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  guidance  in  the  formulation  of 
exploratory  questions  and,  if  necessary, 
transcription  of  the  questionnaire 
responses  to  the  DTS. 

Appraisal  team 

Develops  and  agrees  upon  set  of  follow-up 
questions  for  each  of  the  SE  leaders,  and 
"listen  fors"  during  practitioner  sessions. 

Table  2-13.  Participants  for  Analyze  Questionnaire. 


Four  hours 


continued  on  next  page 
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2.2.3  Analyze  Questionnaire,  Continued 


Steps 


Table  2-14  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  responses 

The  appraisal  team  reviews  the  questionnaires 
and  the  data  tracking  sheets. 

Transcribe  to  DTS 
(if  necessary) 

The  appraisal  team,  with  guidance  from  the 
facilitator,  transcribes  the  questionnaire 
responses  from  the  original  questionnaires  to 
the  DTS.  This  is  most  easily  accomplished 
with  2-person  teams,  with  one  calling  out 
question  number  and  response,  and  the  other 
recording  in  the  appropriate  spot  in  the  DTS. 
With  a  6-person  appraisal  team  (three  2-person 
teams),  all  17  process  areas  can  be  fairly  easily 
transcribed  in  an  hour  (assuming  3-4 
respondents  per  questionnaire).  The 
handwritten  DTSs  can  then  be  transcribed  by 
support  staff  for  future  electronic  processing. 

Generate  questions 

The  appraisal  team,  with  guidance  from  the 
facilitator,  generates  20-40  exploratory 
questions  for  each  systems  engineering  lead. 

The  questions  should  be  designed  to  elicit  more 
then  just  a  yes/no  response.  For  example, 
questions  often  begin  with,  “Would  you  please 
describe  . . .”  When  looking  for  certain 
responses,  e.g.,  “SE  management  plan,”  note 
these  words  as  “listen  fors"  which  the  facilitator 
can  use  as  a  cue  to  ask  additional  questions  if 
they  are  not  mentioned.  Some  questions  may 
be  accompanied  by  a  request  for  relevant  or 
supporting  documents. 

Questions  should  be  used  to  refine  answers  or 
explore  inconsistencies.  There  is  a  separate  set 
of  questions  for  each  systems  engineering  lead. 
However,  there  is  usually  some  overlap.  Once 
the  questions  are  developed  they  should  be 
transcribed  into  an  appropriate  form,  and  copies 
should  be  made  for  each  team  member.  The 
form  should  list  each  question,  any  “listen 
fors,"  document  requests,  and  room  for  notes. 

Table  2-14.  Steps  for  Analyze  Questionnaire. 


continued  on  next  page 
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2.2.3  AndlyzG  QuGStionn3irG,  continued 


Steps,  continued 


Tailorable 

parameters 

Exit  criteria 

Notes 


Step 

Guidance 

Generate  questions, 
continued 

If  the  exploratory  question  generation  process 
is  not  well  managed,  it  can  easily  exceed  the 
typical  four  hour  duration.  A  method  for 
quickly  generating  a  pool  of  questions  follows; 

•  Facilitator  leads  exploratoiy  question 
generation  example  with  one  process  area  to 
get  team  oriented  to  using  the  data  embedded 
in  the  questionnaire. 

•  Process  areas  are  split  up  between  the 
appraisal  team  members  according  to  their 
specialty  areas  and  to  provide  balance  in 
workload. 

•  For  each  process  area,  the  team  member  uses 
the  initial  DTS  to  scan  for  inconsistencies 
between  the  projects  and  within  a  project. 

•  Where  probe  points  are  discovered,  the  team 
member  formulates  a  candidate  question  on  a 
post-it  note,  marked  with  team  member's 
initials,  the  SE  lead  (or  leads)  for  whom 
question  is  intended,  and  the  PA  item 
reference  that  led  to  the  question. 

•  The  post-its  can  be  laid  out  in  a  matrix  with 
PAs  across  the  top  and  SE  leads  down  the 
side  to  get  a  picture  of  the  sampling  space 
represented  by  the  initial  question  pool. 

•  The  team  reconvenes  and  comes  to  consensus 
on  which  items  to  ask  each  SE  lead. 

Table  2-14.  Steps  for  Analyze  Questionnaire,  continued 


•  When  analysis  is  performed  (may  be  done  before  on-site  week). 


•  Exploratory  questions  and  data  requests  are  prepared  for  each  systems 
engineering  lead. 


The  facilitator  can  prepare  for  this  step  before  the  on-site  period.  A  team 
may  request  and  use  documentation  generated  for  or  by  a  process  area  to 
help  demonstrate  a  capability  level,  in  addition  to  the  interview  data. 
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2.2.4  Interview  Systems  Engineering  Leads 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


2.2.4  Interview  Systems  Engineering  Leads  (IL) 


The  purpose  of  Interview  Systems  Engineering  Leads  is  to  resolve  any 
misunderstandings  about  the  responses  to  the  questionnaire,  and  to 
explore  areas  that  the  appraisal  team  wishes  to  have  clarified. 


The  session  facilitator  introduces  the  team,  explains  the  purpose  of  the 
session,  and  asks  the  systems  engineering  lead  the  exploratory 
questions.  A  session  recorder  tracks  the  responses  and,  along  with  the 
rest  of  the  appraisal  team,  takes  notes.  As  a  result  of  some  responses, 
the  systems  engineering  lead  may  be  asked  to  supply  certain  documents 
for  later  review.  There  is  a  separate  session  for  each  systems 
engineering  lead. 


Table  2-15  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Session  facilitator 

Conducts  session  and  asks  exploratory 
questions.  The  session  facilitator  need  not  be 
the  appraisal  facilitator.  If  team  members 
possess  the  skills  and  experience  to  conduct 
structured  interviews,  they  should  be 
encouraged  to  conduct  at  least  some  of  the 
interviews  with  the  systems  engineering  leads. 

Session  recorder 

Records  responses,  monitors  the  time,  and 
make  a  list  of  any  requested  documentation. 

Appraisal  team 

Takes  notes  of  responses  and  occasionally  ask 
for  additional  clarification. 

Systems  engineering 
lead 

Responds  to  questions. 

Table  2-15.  Participants  for  Interview  Systems  Engineering 

Leads. 


continued  on  next  page 
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2.2.4  Interview  Systems  Engineering  Leads,  continued 


Typical  duration 
Steps 


Tailorable 

parameters 

Exit  criteria 


One  hour  per  session  +15  minutes  for  team  discussion  per  project 


Table  2-16  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Introduction 

The  session  facilitator  welcomes  the  systems 
engineering  lead  and  introduces  the  appraisal 
team.  The  facilitator  then  explains  the  purpose 
of  the  session  and  how  it  will  be  conducted. 
He/she  also  reminds  the  systems  engineering 
lead  about  the  confidentiality  mles  and 
encourages  frank  responses  to  the  questions. 

Exploratory 

questions 

The  session  facilitator  asks  each  exploratory 
question  while  the  session  recorder  keeps  track 
of  the  time  and  any  requests  for  documents. 

All  other  team  members  take  notes.  Any  team 
member  can  ask  a  question,  though  the  session 
facilitator  should  be  allowed  to  lead  the 
questioning  to  ensure  that  all  high-priority 
questions  are  covered.  The  session  facilitator 
should  follow-up  on  the  “listen  fors"  and  make 
any  document  requests. 

Closing 

At  the  end  of  the  questions  (or  when  time  has 
expired),  the  session  facilitator  concludes  the 
session  by  reminding  the  systems  engineering 
lead  of  the  time  and  location  of  their  next 
meeting  for  reviewing  the  preliminary  findings. 
The  session  recorder  reminds  the  systems 
engineering  lead  of  any  document  requests  and 
arranges  for  collecting  them.  The  session 
facilitator  thanks  the  systems  engineering  lead 
for  his/her  cooperation. 

Table  2-16.  Steps  for  interview  Systems  Engineering  Leads. 


•  Who  is  assigned  as  the  session  facilitator. 

•  Depth  of  follow-up  on  individual  issues. 


•  As  many  questions  as  possible  from  interview  script  are  covered. 


continued  on  next  page 
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2.2.4  Interview  Systems  Engineering  Leads,  continued 


Notes 


I 


The  reason  interview  scripts  are  ordered  by  priority  is  to  make  sure  that 
the  most  important  questions  (to  the  team)  are  asked.  It  is  uncommon 
for  all  the  scripted  questions  to  be  addressed.  A  team  may  request  and 
use  documentation  generated  for  or  by  a  process  area  to  demonstrate  a 
capability  level. 

Depending  on  the  acceptability  in  the  appraisal  organization’s  culture,  it 
is  very  useful  to  have  the  session  recorder  use  a  laptop  or  other 
workstation  to  key  in  responses  and  dialogue  ensuring  from  EQs  as  they 
are  happening.  If  the  EQs  have  been  prepared  on  line,  and  responses 
are  recorded  in  spaces  between,  this  permits  much  more  rapid 
consolidation  post-interviews.  More  than  one  or  two  laptops  being  used 
seems  daunting  to  most  interviewees,  however,  and  manual  note  taking 
is  recommended  for  most  team  members.  Ask  before  using  a  laptop  or 
other  recording  instrument. 
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2.2.5  Consolidate  Data  from  SE  Leads 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.5  Consolidate  Data  from  SE  Leads  (CDl) 


The  purpose  of  Consolidate  Data  from  SE  Leads  is  to  assimilate  the 
notes  taken  during  the  interviews  with  the  systems  engineering  leads. 


Consolidate  Data  from  SE  Leads  involves  updating  the  DTS  to  reflect 
the  information  gained  in  the  interviews  with  the  systems  engineering 
leads.  It  also  allows  the  members  of  the  appraisal  team  to  verify  their 
understanding  of  the  information  obtained  in  the  interviews  with  the 
other  team  members.  Finally,  this  data  consolidation  step  allows  the 
team  to  strategize  any  needed  changes  in  the  scheduling  or  other  aspects 
of  the  remaining  data  gathering. 


Table  2-17  lists  the  primary  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  guidance  and  model  expertise  for  the 
team's  deliberations. 

Appraisal  team 

Review  their  notes,  discuss  any  issues, 
formulate  observations,  and  update  their  data 
tracking  sheets 

Table  2-17.  Participants  for  Consolidate  Data  from  SE  Leads. 


One  to  1 .5  hours 


continued  on  next  page 
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2.2.5  Consolidate  Data  from  SE  Leads,  Continued 


Steps 


Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-18  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  notes 

Each  team  member  privately  reviews  his/her 
notes  from  the  interviews  with  the  systems 
engineering  leads. 

Update  DTS 

Each  team  member  updates  the  data  tracking 
sheets  by  adding  a  plus  (+)  or  minus  (-)  to 
those  base/generic  practices  for  which 
corroborating  or  opposing  evidence  was  heard. 

Discuss  issues 

Team  members  raise  issues  that  they  have 
encountered  during  the  review  and  update 
steps.  The  facilitator  moderates  the  discussions 
to  give  each  issue  a  chance  to  be  aired.  At  the 
end  of  the  discussion,  team  members  may  wish 
to  alter  their  entries  to  their  data  tracking  sheets. 

Preparation 

The  team  discusses  the  upcoming  events  and 
any  alterations  that  they  may  wish  to  make 
based  on  the  previous  discussions.  In 
particular,  the  team  may  want  the  session 
facilitators  to  guide  the  discussions  into  certain 
areas  during  one  or  more  of  the  practitioner 
interviews.  This  direction  should  be  carefully 
limited,  as  too  much  direction  of  the  practitioner 
discussions  will  stifle  spontaneity. 

Table  2-18.  Steps  for  Consolidate  Data  from  SE  Leads. 


•  None 


•  Team  members  complete  update  of  their  DTS. 

•  Changes  to  schedule  and  other  data  gathering  mechanisms  are  agreed 
to. 


See  Appendix  C  for  details  on  the  format  and  suggested  use  of  the  data 
tracking  sheets. 
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2.2.6  Interview  Practitioners 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.6  Interview  Practitioners  (IP) 


The  purpose  of  Interview  Practitioners  is  to  meet  with  the  actual  systems 
engineers  and  practitioners  of  related  support  processes  (e.g.,  quality 
assurance,  manufacturing)  to  obtain  corroboration  of  the  key  issues 
previously  asserted  and  to  identify  new  issues. 


Interview  Practitioners  involves  facilitating  a  relatively  free-form 
discussion  centered  around  the  question,  "What  works  or  doesn't  work 
well  in  the  systems  engineering  process?"  This  discussion  typically 
provides  corroborating  and  clarifying  data  in  relation  to  other  sources. 

In  some  cases,  the  facilitator  may  provide  minimal  direction  into  topic 
areas  needed  to  be  “filled  in,”  based  on  the  DTS.  It  is  usually  best  to  let 
the  free-form  part  of  the  discussion  begin  the  session;  otherwise, 
participants  may  get  the  impression  of  a  question  and  answer  format, 
and  cease  volunteering  the  issues  that  are  important  to  them. 


Table  2-19  lists  the  primary  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Session  facilitator 

Keeps  discussion  moving  without  influencing  its 
direction. 

Appraisal  team 

Takes  notes  of  discussions. 

Practitioners 

Present  issues  from  their  perspective. 

Table  2-19.  Participants  for  Interview  Practitioners. 


Six  hours  (3  hours  and  45  minutes,  with  15  minute  breaks) 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  continued 


Steps  Table  2-20  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Preparation 

Arrange  the  interview  area  before  the  practitioners' 
arrival.  The  setup  can  be  around  a  table  or  in  a 
circle  without  a  table.  In  either  case,  the  team 
should  occupy  every  other  seat,  forcing  the 
practitioners  to  spread  out  around  the  circle.  The 
team  members  should  be  in  their  seats  before  the 
practitioners  arrive.  It  is  important  to  begin  the 
meeting  on  time.  If  possible,  have  someone  not  on 
the  team  call  any  latecomers  or  no-shows. 

Introduction 

The  session  facilitator 

•  Welcomes  the  practitioners  and  introduces  the 
session's  topic  (if  appropriate). 

•  Explains  the  purpose  of  the  session  and  how  it 
will  be  conducted,  i.e.,  an  unstructured 
discussion  of  the  systems  engineering  process 
from  the  viewpoint  of  the  practitioners. 

•  Lets  the  practitioners  know  that  the  team  will 
listen  and  take  notes  but  that  the  direction  of  the 
discussions  will  be  up  to  them. 

•  Warns  them  that,  at  the  end  of  the  session,  each 
participant  will  be  asked  for  a  strength  and  a 
weakness  of  the  appraised  entity. 

The  session  facilitator  reminds  the  practitioners 
about  the  confidentiality  rules,  encourages  frank 
discussions,  and  cautions  the  participants  about 
repeating  comments  outside  of  the  interview.  The 
team  and  practitioners  then  introduce  themselves, 
giving  their  name  and  role  (team  members  should 
identify  themselves  as  "members  of  the  appraisal 
team").  The  facilitator  then  turns  the  meeting  over 
to  the  practitioners. 

Table  2-20.  Steps  for  Interview  Practitioners. 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  continued 


Steps,  continued 


Step 

Guidance 

Discussion 

The  session  facilitator  keeps  the  conversation 
moving  with  as  little  intrusion  as  possible.  Team 
members  should  not  participate  in  the 
conversations.  The  session  facilitator  should 
note  non-participating  practitioners  and  try  to 
bring  them  into  the  conversation  in  a  non-directed 
manner  (e.g.,  "What  do  you  think  about  that, 
Tom?"). 

If  the  discussions  lag,  or  special  topics  have  been 
identified,  the  session  facilitator  should  gently 
"nudge"  the  discussion  in  the  desired  direction. 

If  this  is  not  done  carefully,  the  participants  will 
continually  look  to  the  facilitator  for  direction  and 
the  interview  will  loose  its  spontaneity.  The 
facilitator  should  use  leading  comments  (e.g., 
"How  about  quality  assurance  . .  .?"),  and  avoid 
direct  yes/no  questions  (e.g.,  "Do  you  do  quality 
assurance?"),  or  judgmental  questions  (e.g., 

"Why  don't  you  do  quality  assurance?"). 

Table  2-20.  Steps  for  Interview  Practitioners,  continued 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  continued 


Steps,  continued 


Tailorable 

parameters 


Exit  criteria 


Step 

Guidance 

Closing 

Near  the  end  of  the  session,  the  facilitator  brings 
the  conversation  to  an  end  and  asks  two 
questions  of  each  practitioner.  If  the  facilitator 
senses  that  the  discussions  have  reached  the  end 
of  their  usefulness,  he/she  can  end  the  session 
before  the  end  of  the  full  time  period. 

The  session  facilitator  then  asks  each  practitioner 
the  following  question: 

If  you  could  change  one  thing  in  your 
organization  other  than  your  boss  or  your 
paycheck,  what  would  it  be  ? 

Next,  the  facilitator  asks  each  practitioner 
(perhaps  going  around  the  group  in  the  opposite 
direction): 

Other  than  the  people,  what  do  you  think 
is  this  organization's  major  strength? 

Note  the  word  "organization"  should  be 
customized  for  the  situation. 

Finally,  the  session  facilitator  thanks  the 
practitioners  for  their  participation  and  reminds 
them  of  the  time  and  location  for  presenting  the 
draft  findings. 

Process  check 

In  the  interval  between  practitioner  interviews, 
the  team  should  perform  a  brief  process  check  of 
how  the  last  inter/iew  went,  and  discuss  any 
changes  or  special  situations  anticipated  with  the 
next  group.  The  sessions  are  long,  so  be  sure  to 
leave  time  for  breaks  before  the  next  group 
arrives. 

Table  2-20.  Steps  for  Interview  Practitioners,  continued 


♦  Number  and  makeup  of  practitioner  groups. 

•  Use  of  middle  manager  or  other  focus  groups,  perhaps  outside  of  on¬ 
site  week. 


•  Session  completed. 


continued  on  next  page 
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2.2.6  Interview  Practitioners,  Continued 


Notes 


Practitioner  groups  may  be  grouped  according  to  life-cycle  phase 
(requirements  engineers/architects  in  one  group,  integrators/testers  in 
another),  organizational  boundaries  (all  the  CM  and  training  folks 
together  since  they  are  outside  the  systems  engineering  organization 
hierarchy),  or  other  ways  that  will  maximize  the  information  obtained. 

A  project  management  practitioner  group  might  be  set  up  if  the  systems 
engineering  and  project  management  functions  (as  defined  in  the  project 
section  of  the  SE-CMM)  are  across  organizational  boundaries.  The 
point  is  to  use  the  particular  organizational  context  to  best  advantage  to 
corroborate  data  obtained  via  the  questionnaire  and  interviews  with  the 
systems  engineering  leads.  A  team  may  request  and  use  documentation 
generated  for  or  by  a  process  area  to  demonstrate  a  capability  level. 

Middle  manger  buy-in  may  be  obtained  by  having  a  practitioner  or  focus 
group  before  the  on-site  week,  which  provides  an  opportunity  to  engage 
them  as  "owners"  of  the  action  plan  recommendations  after  the  on-site 
period.  In  addition,  in  some  organizations,  marketing  or  business 
developrnent  will  "own"  product  line  evolution,  and  should  be  included 
in  practitioner  groups  or  have  a  mini-focus  group. 

A  session  recorder  may  use  a  laptop  or  other  workstation  if  the 
organization's  culture  permits  it.  This  greatly  facilitates  consolidation  of 
data  and  application  of  key  phrases  to  findings.  Ask  before  using  a 
laptop  or  other  recording  equipment.  In  no  case  should  more  than  one 
or  two  use  laptops. 
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2.2.7  Consolidate  Data  from  Practitioners 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.7  Consolidate  Data  from  Practitioners  (CD2) 


The  purpose  of  Consolidate  Data  from  Practitioners  is  to  assimilate  the 
notes  taken  during  the  practitioner  interviews  and  form  preliminary 
ratings  for  each  process  area. 


Consolidate  Data  from  Practitioners  involves  updating  the  DTS  to  reflect 
the  information  gained  in  the  interviews  with  the  systems  engineering 
leads.  It  also  allows  the  members  of  the  appraisal  team  to  verify  their 
understanding  of  the  information  obtained  in  the  interviews  with  the 
other  team  members.  Finally,  this  data  consolidation  step  allows  the 
team  to  strategize  any  needed  changes  in  the  scheduling  or  other  aspects 
of  the  remaining  data  gathering.  In  addition.  Consolidate  Data  from 
Practitioners  is  the  first  time  when  the  team  attempts  to  develop  a  rating 
profile. 


Table  2-21  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Provides  guidance  and  model  expertise  for  the 
team's  deliberations. 

Appraisal  team 

Review  their  notes,  discuss  any  issues,  formulate 
observations,  update  their  data  tracking  sheets, 
and  formulate  preliminary  ratings. 

Table  2-21.  Participants  for  Consolidate  Data  from  Practitioners. 


One  to  two  hours 


continued  on  next  page 
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2.2.7  Consolidate  Data  from  Practitioners,  Continued 


Step 


Table  2-22  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  notes 

Each  team  member  privately  reviews  his/her 
notes  from  the  practitioner  interviews, 
committing  issues  to  post-its. 

Update  DTS 

Each  team  member  updates  his/her  data  tracking 
sheets  by  adding  a  plus  (-1-)  or  minus  (-)  to 
those  base/generic  practices  for  which 
corroborating  or  opposing  evidence  was  heard. 

Discuss  issues 

Team  members  raise  issues  that  they  have 
encountered  during  the  review  and  update 
steps.  The  facilitator  moderates  the  discussions 
to  give  each  issue  a  chance  to  be  aired.  At  the 
end  of  the  discussion,  team  members  may  wish 
to  alter  their  entries  to  their  data  tracking  sheets. 

Form  preliminary 
ratings 

Based  on  their  data  tracking  sheets,  the 
appraisal  team  forms  a  preliminary  rating  for 
each  process  area.  A  capability  level  is 
considered  achieved  if,  in  the  opinion  of  the 
team  members,  100%  of  the  base/generic 
praetices  of  that  level  are  performed.  To 
achieve  level  1,  100%  of  the  base  practices, 
across  the  appraisal  entity  as  a  whole,  must  be 
performed.  For  levels  two  2  through  5,  50% 
of  the  generic  practices  associated  with  that 
level  must  be  performed  across  the 
organization. 

It  is  not  critical  that  the  team  achieve  consensus 
at  this  time;  however,  plans  must  be  made  for 
resolving  any  conflicts.  In  this  case, 
preliminary  findings  can  be  crafted  to  explore 
any  open  issues  further. 

Preparation 

The  team  discusses  the  upcoming  events  and 
any  alterations  that  they  may  wish  to  make 
based  on  the  previous  discussions.  In 
particular,  the  team  may  wish  to  add  a  few 
exploratory  questions  or  special  findings  to  the 
preliminary  findings  in  order  to  resolve 
conflicting  evidence,  especially  where  the 
ratings  are  affected. 

Table  2-22.  Steps  for  Consolidate  Data  from  Practitioners. 


continued  on  next  page 
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2.2.7  Consolidate  Data  from  Practitioners,  Continued 


Tailorable 

parameters 

Exit  criteria 

Notes 


•  Initial  rating  profile  is  developed  (may  be  deferred). 


•  DTS  are  updated. 

•  Scheduling  and  data  gathering  changes  are  agreed  to  where 
appropriate. 


See  Appendix  C  for  details  on  the  format  and  suggested  use  of  the  data 
tracking  sheets. 

It  is  recommended  that  each  team  member  copy  issues  (strengths  and 
weaknesses)  to  post-its  as  notes  are  reviewed.  The  affected  PA  and 
team  member's  initials  should  be  included  together  with  the  session  in 
which  the  issue  was  recorded.  A  typical  team  member  generates  30- 
100+  issues,  and  can  generate  250-400  total. 
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2.2.8  Develop  Preliminary  Findings 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


2.2.8  Develop  Preliminary  Findings  (PF) 


The  purpose  of  Develop  Preliminary  Findings  is  to  formulate  a  set  of 
findings  that  reflect  an  initial  synthesis  of  the  accumulated  data  from  all 
data  sources  used  in  the  appraisal. 


During  Develop  Preliminary  Findings,  the  appraisal  team  systematically 
analyzes  the  data  from  all  sources  to  generate  a  list  of  preliminary 
findings  related  to  the  process  areas  and  capability  levels  under 
investigation. 


Table  2-23  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Leads  team  in  brainstorm  activities  and 
oversees  the  preparation  of  the  preliminary 
findings. 

Appraisal  team 

Synthesize  the  accumulated  data  into 
preliminary  findings,  including  both  strengths 
and  weaknesses. 

Table  2-23.  Participants  for  Develop  Preliminary  Findings. 


continued  on  next  page 
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2.2.8  Develop  Preliminary  Findings,  continued 


Steps 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-24  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Individual  appraisal 
team  members  record 
their  candidate 
preliminary  findings 
according  to  process 
areas/common 
features 

An  easy  way  to  record  candidate  findings  is  to 
record  them  on  post-it  notes  along  with  the 
process  area  or  common  feature  tag  and  author 
initials.  You  can  show  their  affinities  by 
placing  the  notes  either  on  a  long  table  or  on 
multiple  flip  charts  (one  per  PA). 

Mini-teams  collate  the 
findings,  eliminate 
redundancies,  and 
look  for  common 
threads 

Teams  of  two  work  well  to  narrow  the  initial 
set  by  clustering  to  a  number  that  the  entire 
team  can  review.  The  teams  can  move  to  the 
next  process  area  in  line  after  they  are  done 
with  their  first  one,  and  so  on  until  all  are 
clustered. 

Mini-teams  present 
candidates  to  entire 
team  for  consensus  on 
findings  to  put  forth 

Consensus  here  is  more  of  the  nature  of,  "Can  I 
live  with  the  systems  engineering  team  leads' 
agreement  or  disagreement?"  rather  than  "I  can 
agree  with  this."  Remember,  not  all  the 
findings  will  be  validated  by  the  feedback 
sessions. 

Preliminary  findings 
are  recorded  for  use  in 
the  feedback  sessions 
for  the  next  day 

We  recommend  that  you  use  a  flat  file  database, 
such  as  a  word  processor,  to  record  the 
findings  according  to  process  area/common 
feature  with  a  form  that  allows  room  for  notes 
from  the  feedback  session.  This  makes  it  easy 
for  the  team  to  record  notes  the  following  day. 

Table  2-24.  Steps  for  Develop  Preliminary  Findings. 


Four  or  more  hours 


•  Level  of  granularity  of  preliminary  findings. 

•  Number  of  preliminary  findings  presented  for  validation. 


•  40-60  preliminary  findings  that  can  be  verified  with  the  systems 
engineering  leads. 


One  PA  per  flipchart  is  recommended  with  an  additional  flipchart  each 
for  "strengths"  and  "not  in  model"  issues. 
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2.2.9  Review  Preliminary  Findings 


Element  title/tag 
Purpose 

Summary 

description 

Major 

participants 


Typical  duration 


2.2.9  Review  Preliminary  Findings 


The  puipose  of  Review  Preliminary  Findings  is  to  obtain  feedback  on 
the  preliminary  findings  from  the  systems  engineering  leads  who 
provided  the  original  data. 


Review  Preliminary  Findings  involves  feeding  back  the  synthesized  data 
in  the  form  of  strengths  and  weaknesses  to  the  same  systems 
engineering  leads  who  were  the  original  sources  of  the  data. 


Table  2-25  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Session  facilitator 

Presents  preliminary  findings  to  the  systems 
engineering  leads  for  their  comment.  If 
possible,  the  session  facilitator  role  should  be 
filled  by  the  appraisal  team  leader  to  increase 
his/her  comfort  with  explaining  the  findings. 

Session  recorder 

Tracks  session  and  monitors  time.  If  the  team 
leader  is  the  session  facilitator,  the  trained 
facilitator  should  act  as  the  session  recorder. 

Systems  engineering 
lead 

Provides  feedback  on  preliminary  conclusions. 

Appraisal  team 

Takes  notes. 

Table  2-25.  Participants  for  Review  Preliminary  Findings. 


One  hour  per  project 


continued  on  next  page 
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2.2.9  Review  Preliminary  Findings,  continued 


Steps  Table  2-26  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Introduction 

The  session  facilitator  welcomes  the  systems 
engineering  lead  and  describes  what  the  team  has 
done  since  the  last  meeting.  He/she  describes  the 
purpose  of  this  session  and  explains  that  the 
findings  are  preliminary  and  will  likely  change; 
therefore,  it  is  especially  important  not  to  discuss 
this  version  of  the  findings  outside  of  this  review. 

The  facilitator  also  tells  the  lead  that  at  the  end  of 
the  session  he/she  will  ask  for  a  strength  and  an 
important  change  in  the  appraised  entity. 

Review  findings 

The  session  facilitator  presents  each  preliminary 
finding  to  the  systems  engineering  lead  and  asks 
whether  he/she  believes  that  the  finding  is  true  on 
their  project  and  whether  he/she  believes  it  is 
generally  true  for  the  appraised  entity. 

The  facilitator  or  members  of  the  team  can  ask  for 
clarification  or  a  follow-on  question.  However, 
all  preliminary  findings  need  to  be  presented  in  the 
allocated  time.  Therefore,  the  facilitator  and 
recorder  need  to  keep  a  tight  rein  on  the  time  and 
not  let  follow-up  questions  jeopardize  the 
schedule. 

Table  2-26.  Steps  for  Review  Preliminary  Findings. 


continued  on  next  page 
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2.2.9  Review  Preliminary  Findings,  Continued 


Steps,  continued 


Step 

Guidance 

Closing 

The  session  facilitator  closes  the  session  by  asking 
the  systems  engineering  lead  the  following 
question; 

If  you  could  change  one  thing  in  your 
organization  other  than  your  boss  or  your 
paycheck,  what  would  it  be? 

Next  the  facilitator  asks; 

Other  than  the  people,  what  do  you  think  is 
this  organization’s  major  strength? 

Note  the  word  "organization"  should  be 
customized  for  the  situation. 

Finally,  the  session  facilitator  thanks  the  systems 
engineering  lead  for  participating  and  reminds 
him/her  of  the  time  and  location  for  presenting  the 
draft  findings. 

Process  check 

In  the  interval  between  interviews,  the  team  should 
perform  a  brief  process  check  of  how  the  last 
interview  went,  and  discuss  any  changes  or 
special  situations  anticipated  with  the  next  systems 
engineering  lead.  Again,  be  sure  to  leave  time  for 
a  visit  to  the  bathroom  and  still  leave  time  to  be 
back  in  the  room  before  the  next  lead  arrives. 

Table  2-26.  Steps  for  Review  Preliminary  Findings,  continued 


Tailorable 

parameters 


•  Documentation  may  be  requested  and  reviewed. 

•  SE  leads  may  be  provided  a  table  of  preliminary  findings  and  check 
yes/no  for  project  and  organization  on  paper;  the  session  facilitator 
then  asks  if  there  are  any  questions. 

•  SE  leads  may  review  preliminary  findings  in  parallel  sessions, 
providing  teams  of  two  (facilitator  and  recorder)  attend  each. 


Exit  criteria  •  Feedback  on  the  applicability  of  the  preliminary  findings  has  been 

obtained. 

•  Potential  modifications  to  findings  are  noted  by  the  team. 


Notes  None. 
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2.2.10  Develop  Draft  Rating 


Element  title/tag 


Purpose 


Summary 

description 


Major 

participants 


Typical  duration 


2.2.10  Develop  Draft  Rating  (DR) 


The  purpose  of  Develop  Draft  Rating  is  to  assimilate  the  notes  taken 
during  the  review  of  preliminary  findings  and  to  formulate  a  draft  set  of 
ratings  for  each  process  area. 


Develop  Draft  Rating  involves  preparing  to  synthesize  the  preliminary 
findings  into  a  set  of  draft  findings  which  form  the  core  of  the  findings 
briefing  to  be  presented  to  the  appraisal  participants.  In  addition,  this  is 
the  point  at  wWch  the  appraisal  team  must  reach  consensus  on  the  draft 
rating  profile. 


Table  2-27  lists  the  primary  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Facihtator 

Provides  guidance  and  model  expertise  for  the 
team's  deliberations. 

Appraisal  team 

Review  their  notes,  discuss  any  issues,  formulate 
observations,  update  their  data  tracking  sheets, 
and  formulate  draft  ratings. 

Table  2-27.  Participants  for  Develop  Draft  Rating. 


One  to  two  hours 


continued  on  next  page 
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2.2.10  Develop  Draft  Rating,  continued 


Steps 


Table  2-28  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  notes 

Each  team  member  privately  reviews  his/her 
notes  from  the  interview  with  the  systems 
engineering  leads. 

Update  DTS 

Each  team  member  updates  his/her  data  tracking 
sheets  by  adding  a  plus  (+)  or  minus  (-)  to 
those  base/generic  practices  for  which 
corroborating  or  opposing  evidence  was  heard. 

Discuss  issues 

Team  members  raise  issues  that  they  have 
encountered  during  the  review  step.  The 
facilitator  moderates  the  discussions  to  give 
each  issue  a  chance  to  be  aired.  At  the  end  of 
the  discussion,  team  members  may  wish  to  alter 
the  entries  on  their  data  tracking  sheets. 

Draft  ratings 

Based  on  their  data  tracking  sheets,  the 
appraisal  team  forms  a  preliminary  rating  for 
each  process  area.  A  capability  level  is 
considered  achieved  if,  in  the  opinion  of  the 
team  members,  the  base/generic  practices  for 
that  level  are  performed.  To  achieve  level  1, 
100%  of  the  base  practices  must  be  performed. 
For  levels  2-5,  the  generic  practices  associated 
with  that  level  must  be  performed  (but  80% 
may  be  an  acceptable  standard). 

At  this  point  in  the  appraisal  process,  team 
consensus  on  the  ratings  is  necessary.  The 
facilitator  leads  the  process  of  building 
consensus.  In  the  unlikely  event  that 
consensus  cannot  be  achieved,  the  team  must 
develop  a  plan  for  presenting  the  draft  findings 
that  will  allow  final  resolution  of  the  ratings, 
based  on  practitioner  and  SE  lead  input. 

Preparation 

The  team  discusses  the  upcoming  events  and 
any  alterations  that  they  may  wish  to  make 
based  on  the  previous  discussions.  In 
particular,  the  team  may  wish  to  include 
additional  slides  with  the  draft  findings 
briefing. 

Table  2-28.  Steps  for  Develop  Draft  Rating. 


continued  on  next  page 
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2.2.10  Develop  Draft  Rating,  continued 


Tailorable 

parameters 

Exit  criteria 

Notes 


•  This  step  would  not  include  ratings  development  if  the  appraisal 
context  excludes  the  rating  profile  from  the  appraisal  results. 


•  Consensus  is  obtained  on  the  draft  rating  profile. 


See  Appendix  C  for  details  on  the  format  and  suggested  use  of  the  data 
tracking  sheets. 
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2.2.11  Develop  Draft  Findings 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2. 1 1  Develop  Draft  Findings  (DF) 


The  purpose  of  Develop  Draft  Findings  is  to  focus  on  a  subset  of  the 
process  areas  and  to  develop  refined  findings  for  each  process  area 
investigated  in  the  appraisal. 


Develop  Draft  Findings  involves  analyzing  the  preliminary  conclusions 
in  light  of  the  contents  of  the  capabiUty  levels,  determining  the  estimated 
process  capability  for  each  process  area  that  was  investigated,  and 
synthesizing  the  prehminary  findings  into  a  manageable  set  for 
presentation  to  the  sponsor.  The  preliminary  findings  are  synthesized 
from  around  45-50  findings  into  5-10  draft  findings  that  address 
findings,  data,  and  consequences. 


Table  2-29  lists  the  prim^  roles  involved  in  this  process  element  and 
the  summary  of  their  activity  during  this  process. 


Role 

Summary 

Facihtator 

Provides  expertise  on  the  SE-CMM  and  guides 
the  team  in  forming  consensus. 

Appraisal  team 

Forms  consensus  for  the  final  findings  and 
estimated  process  capability  levels. 

Table  2-29.  Participants  for  Develop  Draft  Findings. 


Four  or  more  hours 


continued  on  next  page 
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2.2.11  Develop  Draft  Findings,  Continued 


Steps  Table  2-30  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  data 

The  team  should  review  the  draft  ratings  and 
the  systems  engineering  leads'  responses  to  the 
preliminary  findings.  This  information  should 
allow  the  team  to  identify  areas  for  findings. 

The  process  areas  are  a  good  starting  point,  but 
the  appraisal  is  not  limited  to  findings  that 
exactly  match  the  SE-CMM  process  areas. 

There  are  sometimes  local  issues  that  must  be 
addressed  if  the  appraisal  is  to  have  credibility 
with  the  participants. 

The  number  of  findings  should  be  limited  to  at 
least  five,  but  no  more  than  nine.  Too  many 
findings  will  be  discouraging  and  difficult  to 
address;  too  few  will  encourage  false 
confidence  and  not  provide  a  rich  enough  set  of 
issues  for  planning  process  improvement. 

Initially  the  team  may  identify  more  than  nine 
areas,  with  the  understanding  that  it  will  merge 
or  drop  some  later. 

Prioritize  findings 

There  are  usually  more  than  10  synthesized 
findings  to  start.  Prioritization  should  be  based 
on  business  goals,  if  available,  or  on  the  team's 
consensus  on  the  major  barriers  to 
improvement  in  the  organization. 

Wordsmith 

The  team  should  form  small  groups  (two  to 
three  people)  to  edit  and  wordsmith  the 
findings.  This  is  especially  true  if  they  could 
not  agree  on  the  wording  during  the  previous 
step. 

The  facilitator  should  carefully  consider  the 
make-up  of  these  groups  and  the  assignment  of 
findings.  Sometimes  it  is  a  good  idea  to  place 
antagonists  in  the  same  group  and  give  them  the 
controversial  finding  to  edit.  Other  times, 
neutral  parties  should  do  the  editing.  In 
extreme  cases  both  approaches  might  be  tried. 
This  is  where  the  facilitator  makes  use  of 
his/her  teaming  skills  and  knowledge  of  the 
team. 

Table  2-30.  Steps  for  Develop  Draft  Findings. 


continued  on  next  page 
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2.2.11  Develop  Draft  Findings,  continued 


Steps,  continued 


Step 

Guidance 

Form  consensus 

Finally,  each  finding  is  presented  to  the  group 
for  final  edit  and  approval.  The  team  must 
agree  with  each  finding.  If  team  members  go 
away  from  the  appraisal  without  full 
commitment  to  dl  of  the  findings,  the  other 
participants  will  sense  the  lack  of  consensus 
and  interpret  it  as  a  weakness  in  the  findings. 

This  step  continues  until  consensus  is  achieved. 
If  necessary,  the  team  might  have  to  go  back  to 
the  previous  step  in  an  effort  to  come  to  an 
agreement. 

Prepare  slides 

The  findings  are  placed  on  slides  for 
presentation.  See  Appendix  B  for  a  sample  of 
the  findings  presentation  slides.  The  team 
members  should  be  given  copies  of  the  draft 
findings  for  their  notebook,  and  for  recording 
comments  from  the  draft  presentation.  Do  not 
make  copies  for  the  participants,  as  the  findings 
may  change  after  their  initial  presentation.  The 
appraisal  team  leader  should  take  a  copy  with 
him/her  in  order  to  prepare  for  the  presentation 
the  next  morning. 

Table  2-30.  Steps  for  Develop  Draft  Findings,  continued 


Tailorable  •  Level  of  granularity  of  findings, 

parameters  •  Depth  of  analysis  for  determining  capability  level. 


Exit  criteria  •  Consensus  is  obtained  on  draft  findings. 

•  Draft  findings  briefing  is  completed. 


Notes  Findings  may  be  presented  in  the  context  of  process  area  categories, 

process  areas,  base  practices,  generic  practices,  common  features,  or 
capability  levels,  depending  on  the  appraisal  goals. 


2-50 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


2.2.12  Present  Draft  Findings 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.12  Present  Draft  Findings  (PF) 


The  purpose  of  Present  Draft  Findings  is  to  provide  a  vehicle  for  the 
systems  engineering  leads  and  practitioners  to  validate  that  the 
synthesized  findings  represent  the  information  provided  throughout  the 
on-site  phase  of  the  appraisal. 


Present  Draft  Findings  involves  the  appraisal  team  leader  (ATL) 
presenting  the  synthesized  findings  to  the  systems  engineering  leads  as  a 
group,  and  then  to  all  the  practitioners  as  a  group.  This  provides  the 
appraisal  team  leader  with  an  opportunity  to  "rehearse"  the  briefing,  and 
provides  the  appraisal  participants  with  the  opportunity  to  provide 
feedback  on  the  validity  of  the  information. 


Table  2-31  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team 
leader 

Presents  the  draft  findings;  does  not  present  the 
rating  profile  or  the  introductory  material,  just  the 
summarized  findings;  solicits  comments  from 
participants. 

Appraisal  team 

Takes  notes  on  presentation  for  feedback  to 
appraisal  team  leader;  observes  reactions  of 
participants  to  findings;  takes  notes  on  feedback. 

Appraisal 

participants 

Listen  to  findings  presentation;  provide  feedback 
on  whether  the  appraisal  team  captured  what  is 
happening  in  the  organization. 

Session  recorder 

Records  agreed-upon  changes  to  briefing  on  actual 
slides  directly,  as  well  as  on  a  hard  copy. 

Session  facilitator 

Facilitates  the  feedback  portion  of  the  meeting; 
appraisal  team  leader  may  assume  this  role  if 
desired. 

Table  2-31.  Participants  for  Present  Draft  Findings. 


One  hour  for  systems  engineering  leads  as  a  group 
One  hour  for  all  practitioners  as  a  group 


continued  on  next  page 
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2.2.12  PresGnt  Draft  FindinQS,  Continued 


Steps 


Table  2-32  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Describe  purpose  of 
session 

The  presenter  re-emphasizes  that  the  goal  of  the 
appraisal  is  to  capture  the  "state  of  the  practice" 
in  the  appraised  entity;  this  session  allows  the 
apprais^  team  to  validate  that  they  have 
accurately  captured  the  practitioners'  viewpoint. 
The  presenter  then  asks  to  go  through  entire 
briefing  and  then  come  back  to  questions.  It  is 
not  recommended  to  pass  out  a  copy  of  the 
briefing  at  this  time;  it  is  subject  to  change,  and 
incorrect  versions  of  the  briefing  could  damage 
the  credibility  of  the  appraisal. 

Present  findings 

Appraisal  team  leader  rehearses  his/her 
presentation  of  the  findings  portion  of  the 
briefing.  This  briefing  should  not  include  the 
introductory  material  or  rating  profile  to  allow 
extra  time  for  feedback.  Not  showing  the 
rating  profile  also  gives  participants  another 
reason  to  attend  the  main  findings  briefing. 
During  this  briefing,  the  presenter  tries  to  notice 
the  reactions  he/she  gets  from  different 
findings. 

Solicit  feedback 

After  the  run-through  of  the  findings,  the 
appraisal  team  leader  solicits  feedback  from  the 
practitioners,  e.g.,  "Is  this  what  you  told  us 
and  is  this  worded  in  a  way  to  get  positive 
action  from  management?"  The  ATL  facilitates 
the  discussion,  and  when  changes  are  proposed 
by  a  member  of  the  practitioner's  group,  the 

ATL  tries  to  get  a  sense  of  the  agreement  within 
the  group;  there  will  not  be  time  to  get  a  full 
consensus,  just  tiy  to  make  sure  that  the 
comment  is  one  that  engenders  general 
agreement.  The  appraisal  team  is  not  looking 
for  a  lot  of  changes,  just  to  make  sure  that  the 
tone  of  the  findings  is  able  to  be  acted  upon, 
and  that  the  team  is  not  misrepresenting  the  data 
that  were  gathered.  | 

Table  2-32.  Steps  for  Present  Draft  Findings. 


continued  on  next  page 
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2.2.12  Present  Draft  Findings,  continued 


Steps,  continued 


Tailorable 

parameters 


Step 

Guidance 

Make  draft  changes 

Draft  changes  can  be  made  directly  on  the  actual 
slide,  so  participants  can  see  the  direct  effect  of 
their  input;  they  should  also  be  recorded  on  a 
hard  copy  or  electronic  copy  for  discussion 
among  the  team.  (These  draft  changes  will  be 
validated  in  the  next  process  element.) 

Provide  reminders 

The  ATL  reminds  participants  of  the  time  for 
the  findings  briefing  and  emphasizes  that  this  is 
an  opportunity  to  interact  wiA  the  management 
who  sponsored  the  appraisal.  The  ATL  asks 
the  practitioners  to  keep  findings  to  themselves 
until  after  the  findings  briefing.  This  gives 
sponsors  a  chance  to  respond  to  the  findings 
without  the  "rumor  mill"  interfering. 

Adjourn  meeting 

The  ATL  thanks  participants  for  their 
participation  throughout  the  week. 

Table  2-32.  Steps  for  Present  Draft  Findings,  continued 


•  Systems  engineering  lead  and  practitioner  groups  could  conceivably 
be  combined,  though  this  risks  inhibition  of  the  practitioners  in  some 
environments.  If  a  management  practitioner  group  was  used,  you 
may  want  to  combine  them  with  the  systems  engineering  leads. 

•  Someone  other  than  the  appraisal  team  leader  may  be  selected  as  the 
presenter  for  the  findings  briefing,  or  the  presentation  may  be  split 
between  multiple  presenters.  A  common  split  is  to  have  the  facilitator 
present  the  introduction,  the  appraisal  team  leader  present  the  findings, 
and  the  facilitator  close  with  die  "next  steps."  Presentation  by  the 
appraisal  team  leader  has  the  advantage  of  organizational  "ownership" 
of  the  material;  in  some  environments,  the  facilitator  has  the  advantage 
of  being  the  outside  "independent"  expert.  The  decision  of  who 
presents  the  findings  should  be  based  on  the  team's  consensus  of  who 
will  have  the  most  impact  and  achieve  the  most  momentum  for 
improvement  action. 


continued  on  next  page 
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2.2.12  Present  Draft  Findings,  continued 


Exit  Criteria 


Notes 


•  Presentation  of  draft  findings  complete. 

•  Practitioner  concerns  recorded. 


This  session  is  important  in  establishing  the  credibility  of  the  appraisal 
with  the  practitioners  and  for  providing  practitioners  with  momentum 
toward  change.  It  is  important  to  make  sure  there  are  no  findings  where 
the  systems  engineering  leads/practitioners  say,  "This  is  absolutely  not 
true  —  whatever  gave  you  that  idea?"  or  something  similar.  There 
should  not  be  many  changes  that  come  out  of  these  presentations,  but 
you  should  put  a  couple  in,  just  to  show  the  team's  willingness  to  listen, 
even  if  they  are  minor  details. 
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2.2.13  Adjust  Draft  Findings 


Element  title/tag  2.2.13  Adjust  Draft  Findings  (AD) 


Purpose  The  purpose  of  Adjust  Draft  Findings  is  to  ensure  that  the  final  findings 

briefing  accurately  reflects  the  information  obtained  from  the  participants 
by  refining  the  findings  briefing  based  on  feedback  from  the  appraisal 
participants. 

Summary  Adjust  Draft  Findings  involves  the  appraisal  team  discussing  the 

description  presentation  of  the  draft  findings  to  the  appraisal  participants  and 

coming  to  consensus  on  the  dr^t  changes  that  will  be  incorporated  into 
the  final  findings  briefing.  The  final  findings  briefing  is  updated  to 
reflect  the  accepted  changes.  Also,  at  this  time  any  final  adjustments  to 
the  rating  profile  are  made,  if  appropriate.  The  team  provides  feedback 
on  the  dry-run  to  the  appraisal  team  leader,  who  will  be  presenting  the 
findings,  to  help  him/her  refine  the  delivery  of  the  findings. 


Major  Table  2-33  lists  the  primary  roles  involved  in  this  process  element  and  a 

participants  summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team 

Discusses  the  proposed  changes  and  comes  to 
consensus  on  Aem;  provides  feedback  to  the  ATL 
on  the  delivery  of  the  findings. 

Appraisal  team 
leader 

Accepts  feedback  on  the  presentation  of  the 
findings  and  rehearses  rough  points,  where 
necessary;  ensures  that  consensus  on  the  proposed 
changes  is  achieved. 

Session  recorder 

Incorporates  the  accepted  changes  into  the  final 
findings  briefing. 

Appraisal  support 
personnel 

Prepare  slides  and  enough  copies  of  the  final 
briefing  for  the  sponsor  and  the  appraisal  team, 
plus  a  few  extra  in  case  the  sponsor  wants  extras 
for  his/her  own  distribution. 

Table  2-33. 

Participants  for  Adjust  Draft  Findings. 

continued  on  next  page 
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2.2.13  Adjust  Draft  Findings,  Continued 


Typical  duration  Two  hours  (Usually  an  additional  hour  is  provided  for  lunch,  and  any 

last  minute  cleanup/production  can  be  handled  during  that  time  frame  if 
necessary.) 


Steps 


Table  2-34  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Review  draft 
changes 

Obtain  consensus  on  the  proposed  changes 
provided  by  the  participants.  Not  all  changes 
must  be  accepted;  particularly,  ones  that  do  not 
add  to  the  global  issues  of  the  appraisal  may  not 
be  accepted.  The  team  must  agree  on  the 
changes  before  they  are  made. 

Determine  impact  on 
rating  profile 

Based  on  what  was  heard  in  the  findings 
review,  quickly  review  the  rating  profile  to 
ensure  that  it  is  still  valid  (99%  of  the  time  it 
will  be  fine,  but  you  want  to  make  sure 
findings  stay  consistent  with  the  profile).  If 
necessary,  come  to  consensus  on  rating  profile 
changes. 

Discuss  delivery  of 
findings 

Team  members  may  provide  hints  on  the 
delivery  of  the  findings  to  the  ATL  for 
consideration  in  making  the  message  come 
across  more  clearly  or  simply.  The  ATL  may 
rehearse  alternate  approaches  to  delivering 
findings  that  prove  sensitive  or  more  difficult  to 
articulate. 

Table  2-34.  Steps  for  Adjust  Draft  Findings. 


continued  on  next  page 
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2.2.13  Adjust  Draft  Findings,  continued 


Steps,  continued 


Tailorable 

parameters 


Exit  criteria 


Notes 


Step 

Guidance 

Produce  final 
findings  briefing 

Red-line  changes  are  made  to  the  original  if  red- 
lines  are  being  presented,  or  the  electronic  file 
with  the  briefing  is  updated.  At  this  point, 
ensure  that  all  the  other  pieces  of  the  briefing 
(the  introduction,  rating  profile,  and  next  steps) 
are  incorporated  into  the  final  copy.  Appraisal 
support  personnel  make  the  required  number  of 
slides  and  hard  copies.  Unless  otherwise 
established  ahead  of  time,  only  copies  for  the 
sponsor  and  appraisal  team  are  made. 
Frequently,  sponsors  wish  to  control 
distribution  of  the  findings  briefing.  To 
provide  a  copy  to  those  who  want  one,  while 
still  retaining  a  measure  of  control  over 
distribution,  we  suggest  having  a  sign-in  sheet 
at  the  findings  meeting  with  a  place  to  check  if 
people  want  a  copy  of  the  briefing. 

Table  2*34.  Steps  for  Adjust  Draft  Findings,  continued 


•  The  findings  briefing  may  either  be  revised  electronically  so  a  "fresh" 
copy  is  presented,  or  a  red-lined  copy  of  the  draft  briefing  may  be 
used,  depending  on  the  culture  and  support  resources  available. 
(Doing  a  fresh  copy  implies  in-team  computing  support  which  may  or 
may  not  be  available.) 


•  Ratings  profile  is  finalized. 

•  Final  findings  briefing  is  prepared  for  delivery  to  participants. 


Be  sure  the  ATL  is  comfortable  delivering  the  briefing.  The  two 
rehearsals  provided  by  the  draft  findings  briefing  are  usually  sufficient 
to  reduce  the  nervousness  of  presenting  to  the  sponsor. 
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2.2.14  Present  Final  Briefing 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.14  Present  Final  Briefing  (PB) 


The  purpose  of  Present  Final  Briefing  is  to  provide  the  sponsor  with  the 
agreed-upon  data  from  the  appraisal  and  determine  next  steps  for  use  of 
the  findings  (e.g.,  catalyst  for  process  improvement  effort  or  selection 
of  supplier). 


Present  Final  Briefing  involves  presenting  the  results  of  the  appraisal  to 
the  sponsor  and  usually  to  the  other  appraisal  participants  via  a  briefing 
that  synthesizes  the  data  in  a  non-attributable  form  and  provides 
prioritized  findings. 


Table  2-35  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team 
leader 

Presents  the  findings  and  rating  profile  to  the 
sponsor.  Facilitates  any  discussion  after  the 
briefing. 

Appraisal  team 

Notes  feedback  from  sponsor. 

Sponsor 

Accepts  findings  of  appraisal  from  team; 
prioritizes  actions  to  be  taken  based  on 
appraisal  results. 

Appraisal 

participants 

Listen  to  appraisal  results. 

Table  2-35.  Participants  for  Present  Final  Briefing. 


One  to  two  hours 


continued  on  next  page 
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2.2.14  Present  Final  Briefing,  continued 


Steps 


Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-36  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Coach  sponsor  on 
his/her  expected 
participation  in 
meeting 

Remind  sponsor  that  his/her  reaction  to  the 
findings  will  have  an  impact  on  the  enthusiasm 
with  which  follow  on  actions  are  greeted. 

Remind  all  of 
purpose  of  meeting 

Remind  participants  that  they  have  already  had 
opportunities  for  feedback  -  the  gist  of  this 
session  is  to  summarize  what  has  been  found 
during  the  week. 

Thank  sponsor  and 
participants  for 
cooperation  in  the 
appraisal 

Acknowledge  the  support  personnel  and  all 
those  who  helped  the  appraisal  succeed. 

Present  final  briefing 

The  presenter  goes  through  the  entire  briefing, 
including  process  capability  profile  and  next 
steps;  presenter  asks  for  questions  to  be  held 
until  the  end. 

Open  discussion,  if 
appropriate 

Usually  at  the  end  of  the  briefing,  the  sponsor 
does  a  "thank  you"  message,  provides  initial 
reaction,  and  then  opens  with  his/her  own 
questions  or  opens  the  discussion  to  the  floor. 

Table  2-36.  Steps  for  Present  Final  Briefing. 


•  Number  and  type  of  participants  in  audience. 

•  Level  of  detail  of  findings. 

•  Presenter:  in  some  cases,  having  the  facilitator  present  the  process  of 
coming  to  the  findings  and  the  team  leader  present  the  content  of  the 
findings  may  work  best  (see  tailorable  parameters  in  Present  Draft 
Findings). 


•  Final  briefing  presentation  completed. 

•  Participant  questions  answered,  or  recorded  as  actions. 


This  is  where  the  participants  are  likely  to  test  the  sponsor's 
commitment  to  use  the  results  of  the  appraisal. 
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2.2.15  Brief  Sponsor  (optional) 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  Duration 


2.2.15  Brief  Sponsor  (BS) 


The  purpose  of  Brief  Sponsor  is  to  provide  the  sponsor  with  an 
opportunity  to  ask  questions  privately,  obtain  or  provide  feedback  on 
the  appraisal  process,  and  discuss  next  steps  in  more  detail. 


Brief  Sponsor  involves  the  appraisal  team  members  and  sponsors 
having  an  open  discussion  on  the  results  of  the  appraisal,  the  appraisal 
process,  and/or  the  next  steps,  as  appropriate.  No  confidentiality  rules 
are  abrogated  in  this  meeting. 


Table  2-37  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Asks  sponsor  if  any  clarification  or  other 
information  is  needed,  suggests  follow-on 
assignments. 

Appraisal  team 

Answer  sponsor's  questions,  as  appropriate. 

Sponsor 

Asks  any  questions  not  appropriate  for  a  general 
audience;  makes  follow-up  assignments. 

Table  2-37.  Participants  for  Brief  Sponsor. 


Thirty  minutes  to  one  hour 


continued  on  next  page 
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2.2.15  Brief  Sponsor  (optional),  continued 


Steps 


Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-38  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Ask  sponsor  for 
feedback  on  results 

Often  the  sponsor  has  questions  of  clarification 
that  he/she  prefers  not  to  ask  in  a  large  group; 
even  if  there  are  no  specific  questions,  this  is  a 
good  time  to  gauge  the  sponsor's  reaction  - 
sometimes  the  results  are  right  in  line  with  the 
sponsor's  prior  thinking,  sometimes  they  are  a 
surprise. 

Ask  sponsor  for 
feedback  on  process 

Any  feedback  from  the  sponsor  on  the  appraisal 
process  should  be  included  in  the  lessons 
learned  that  are  returned  to  the  SE-CMM 
maintenance  site. 

Discuss  next  steps 
and  add  detail 

At  this  point  the  assignments  for  follow-on 
work  should  be  finalized.  The  facilitator  may 
provide  some  advice  on  how  long  the  follow¬ 
up  activities  will  be  expected  to  t^e  and  the 
level  of  commitment  required  to  finish  the 
report  and  start  to  develop  an  improvement 
plan. 

Table  2-38.  Steps  for  Brief  Sponsor. 


•  This  is  an  optional  process  element,  so  it  may  be  tailored  out  if 
appropriate. 


•  Sponsor  dismisses  team. 


This  is  an  optional  step,  but  is  frequently  used  as  a  way  to  ensure 
sponsor  follow-through  and  assignment  of  actions.  It  also  provides  the 
sponsor  with  an  opportunity  to  get  a  better  understanding  of  the  findings 
which  he/she  may  not  have  been  comfortable  discussing  in  the  large 
meeting. 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


2-61 


2.2.16  Conduct  Wrap-Up 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 


2.2.16  Conduct  Wrap-Up  (CW) 


The  purposes  of  Conduct  Wrap-Up  are  to  obtain  feedback  from  the 
appraisal  team  on  the  appraisal  process  itself,  provide  an  opportunity  for 
consulting  with  the  facilitator  on  moving  forward  with  the  results,  and 
ensure  that  appraisal  materials  are  properly  accounted  for. 


Conduct  Wrap-Up  involves  obtaining  information  about  what  worked 
and  what  did  not  work  from  the  appraisal  team  members  for  feeding 
back  to  the  SAM  maintainers.  It  also  involves  discussing  and  resolving 
findings  and  assignments  from  the  recommendation  report. 


Table  2-39  lists  the  primaiy  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Facilitates  gathering  data  on  what  worked  and  did 
not  work  during  the  appraisal. 

Appraisal  team 
members 

Provide  input  into  what  worked  and  what  didn't 
work. 

Table  2-39.  Participants  for  Conduct  Wrap-Up. 


One  hour 


continued  on  next  page 
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2.2.16  Conduct  Wrap-Up,  Continued 


Steps 


Tailorable 

parameters 

Exit  criteria 

Notes 


Table  2-40  shows  the  steps  for  this  process  element. 


Step 

Guidance 

Conduct  appraisal 
data  gathering 
session 

Getting  the  team  to  write  "what  worked"  and 
"what  didn't  work"  post-it  notes  provides  an 
easy  way  to  gather  data  and  provide  an 
affmitized  list  of  improvement  suggestions  to 
the  maintenance  site  —  what  did  work  is  just  as 
important,  so  that  successful  elements  of  SAM 
don't  get  lost  in  future  revisions 

Verify  findings  and 
recommendations 

If  there  were  areas  of  disagreement  that  came 
out  as  a  results  of  the  final  briefing,  this  time 
can  be  used  to  resolve  how  the  finding  will  be 
presented  in  the  appraisal  report 

Report  assignments 
detailed 

All  appraisal  team  members  should  understand 
their  commitments  to  appraisal  report  sections 
and  their  deadlines. 

Answer  last-minute 
questions 

Often  appraisal  team  members  have  questions 
about  improvement  plans,  etc.,  that  the 
facilitator  can  spend  some  time  answering 
and/or  providing  references  for. 

Table  2-40.  Steps  for  Conduct  Wrap-Up. 


•  Amount  of  time  spent  in  discussing  lessons  learned. 


•  Appraisal  lessons  learned  are  recorded. 

•  Assignments  are  verified. 


Depending  on  the  team,  there  may  be  resistance  to  the  wrap-up  session 
(everybody  wants  to  go  home  and  get  some  sleep! I!  ...  or  catch  a 
plane,  etc.).  The  facilitator  should  judge  how  in-depth  to  make  this 
session  based  on  the  state  of  the  team.  A  celebration  of  some  sort  is  not 
out  of  place  if  the  team  members  are  available.  (Often,  however,  the 
facilitators  have  planes  to  catch.) 
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2.3  Post-Appraisal 


Element  title/tag  2.3  Post-Appraisal  (PO) 


Purpose  The  purpose  of  the  Post-Appraisal  phase  is  to  finish  preparing  the 

appraisi  reports,  document  lessons  learned  in  the  appraisal,  and 
provide  recommendations  for  action  planning.  Figure  2-4  shows  the 
steps  in  the  Post- Appraisal  phase. 


Diagram 


Figure  2-4.  Diagram  of  Post-Appraisai  Phase. 


continued  on  next  page 
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2.3  Post-Appraisal,  Continued 


Summary 

description 


Typical  duration 

Tailorable 

parameters 


Exit  criteria 


Notes 


Table  2-41  shows  the  steps  for  this  process  element. 


ID 

Activity 

Description 

2.3.1 

Report  lessons 
learned 

Record  lessons  learned,  risks  avoided 
and/or  encountered,  and  suggestions  for 
improving  the  method  to  the  SE-CMM 
maintenance  site. 

2.3.2 

Report  appraisal 
outputs  to  other 
parties 

Identify  appraisal  data  in  a  way  that 
maintains  confidentiality  (i.e.,  "sanitize" 
the  data)  and  provide  to  organization(s) 
gathering  state-of-the-practice  data,  if 
agreed  upon  in  the  preparation  phase. 

2.3.3 

Manage  records 

Properly  dispose  of  intermediate  and 
final  data  generated  by  the  appraisal. 

2.3.4 

Develop  findings- 

recommendation 

report 

Expand  findings  briefing  into  a  report; 
add  recommendations. 

Table  2-41.  Summary  Description  of  Post-Appraisal  Phase. 


Two  to  four  weeks 


•  Reporting  of  appraisal  data  to  other  parties  (e.g.,  SE-CMM 
maintenance  site)  must  be  negotiated  and  agreed  upon  in  the 
preparation  phase. 


•  Lessons  learned  reported  back  to  SE-CMM  maintenance  site. 

•  Appraisal  report  completed  and  distributed  appropriately. 

•  Records  disposed  of  in  accordance  with  the  appraisal  plan. 


One  of  the  typical  traps  in  getting  an  improvement  effort  started  is 
allowing  the  release  of  the  appraisal  report  to  drag  out. 
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2.3.1  Report  Lessons  Learned 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


2.3.1  Report  Lessons  Learned  (LL) 


The  purpose  of  Report  Lessons  Learned  is  to  provide  SAM  maintainers 
with  feedback  on  the  strengths  and  weaknesses  in  the  description  of  the 
method  and  model,  and  provide  opportunities  for  improving  the  method 
and  model. 


Report  Lessons  Learned  involves  describing  the  appraisal  goals  and 
tailoring  decisions  made  by  the  team,  and  recording  any  effects  (positive 
or  negative)  of  either  the  described  or  implemented  method. 


Table  2-42  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Facilitator 

Synthesizes  and  records  lessons  learned  from 
appraisal  participation. 

SAM  maintenance 
site 

Accepts  and  addresses  comments  from 
appraisal  team. 

Table  2-42.  Participants  for  Report  Lessons  Learned. 


Two  hours  (to  write  and  send  lessons  learned  information) 


•  Level  of  detail  of  recommendations. 


•  Lessons  learned  sent  to  SAM  maintenance  site. 


Feedback  received  from  appraisal  teams  will  be  a  primary  input  to 
revisions  and  enhancements  to  SAM. 

Feedback  may  be  as  simple  as  a  set  of  post-it  notes,  or  as  elaborate  as  a 
formal  white  paper. 
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2.3.2  Report  Appraisal  Output  to  Other  Parties 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 

Typical  duration 

Tailorable 

parameters 

Notes 


2.3.2  Report  Appraisal  Output  to  Other  Parties  (RP) 


The  purpose  of  Report  Appraisal  Output  to  Other  Parties  is  to  make 
previously  agreed-upon  data  from  the  appraisal  available  to  appropriate 
parties  (e.g.,  the  SE-CMM  maintainers)  for  inclusion  in  state-of-the- 
practice  or  other  appropriate  publications. 


Report  Appraisal  Output  to  Other  Parties  involves  sanitizing  the 
appraisal  data  in  preparation  for  making  previously  agreed-upon  data 
from  the  appraisal  available  to  appropriate  parties  (e.g.,  the  SE-CMM 
maintainers)  for  inclusion  in  state-of-the-practice  or  other  appropriate 
publications. 


Table  2-43  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  the  process. 


Role 

Summary 

Appraisal  team 

Sanitizes  data  and  provides  it  to  agreed-upon 
parties. 

Table  2-43.  Participants  for  Report  Appraisal  Output  to  Other 

Parties. 


Two  to  four  hours 


•  Level  of  detail  provided. 


The  data  to  be  provided  are  negotiated  during  the  preparation  phase. 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


2-67 


2.3.3  Manage  Records 


Element  title/tag 
Purpose 

Summary 

description 


Major 

participants 


Typical  duration 

Tailorable 

parameters 

Exit  criteria 

Notes 


3.3  Manage  Records  (MR) 


The  purpose  of  Manage  Records  is  to  verify  that  ail  records  of  the 
appraisal,  both  intermediate  and  final,  are  disposed  of  in  accordance 
with  agreed-upon  appraisal  ground  rules. 


Manage  Records  involves  reviewing  all  the  gathered  appraisal  materials 
and  ensuring  their  proper  disposal.  Most  intermediate  notes  and  work 
products  are  appropriately  destroyed  once  the  findings  and 
recommendations  report  is  complete.  Notes  may  be  kept  until  then  to 
provide  context  for  recommendations  and  to  verify  the  accuracy  of  the 
report. 


Table  2-44  lists  the  primary  roles  involved  in  this  process  element  and  a 
summary  of  their  activity  during  this  process. 


Role 

Summary 

Appraisal  team 

Gathers  notes  and  other  records  and 
appropriately  disposes  of  them. 

Appraisal  team 
leader 

Secures  all  records  that  are  being  kept. 

Table  2-44.  Participants  for  Manage  Records. 


One  to  two  hours 


•  What  records  are  kept. 

•  How  records  are  secured. 


•  Records  being  kept  are  secured. 

•  Records  not  needed  are  destroyed. 


For  the  records  being  kept,  the  security  provided  should  be  equivalent  to 
that  for  salary  records  or  personnel  actions.  It  is  important  that  the 
integrity  of  the  team  be  maintained;  violation  of  the  stated  confidentiality 
rules,  even  after  the  fact,  can  damage  future  appraisal  activities. 
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Chapter  3:  Guidance  on  Important  SAM 
Processes  for  Initiating  an  Appraisal 


Purpose 


Introduction 


Facilities  support 


Additional 

appraisal 

guidance 


The  purpose  of  this  chapter  is  to  provide  readers  who  are  contemplating 
initiating  a  SAM  with  information  that  will  help  them  properly  resource 
and  prepare  for  the  appraisal. 


The  preparation  phase  is  typically  initiated  six  to  eight  weeks  prior  to  the 
on-site  week.  The  primary  activities  performed  during  the  preparation 
phase  are 

•  Establish  appraisal  objectives. 

•  Determine  boundaries  to  be  placed  on  the  organization  for  the  purpose 
of  establishing  the  scope  of  the  entity  to  be  appraised. 

•  Select  projects,  or  major  aspects  of  a  project,  that  will  receive  the 
questionnaires. 

•  Select  appraisal  team  and  site  coordinator. 

•  Select  systems  engineering  leads,  support  functions,  and  practitioner 
groups. 

•  Obtain  facilities  and  support  material. 

•  Coordinate  schedules,  particularly  senior  management. 

•  Administer  questionnaire. 

A  checklist  to  assist  the  site  coordinator  is  provided  in  Appendix  F. 
Many  of  these  issues  require  guidance  to  optimize  the  organization's  use 
of  SAM.  This  chapter  provides  some  of  that  guidance. 


The  on-site  appraisal  phase  is  very  intensive  for  the  appraisal  team.  To 
maximize  their  efficiency,  it  is  recommended  that  a  large  conference 
room  which  can  be  secured  be  made  available  for  the  entire  week.  Other 
critical  support  facilities  include  computers,  printers,  display  capability, 
and  reproduction  capability.  Recommended  support  facilities  include 
hotel  rooms,  food,  and  secretarial  support. 


All  three  appraisal  phases  are  discussed  in  detail  in  Chapter  2,  "Process 
Element  Summaries."  In  each  process  element,  guidance  is  provided  to 
help  appraisers  avoid  typical  pitfalls  associated  with  that  process 
element. 


continued  on  next  page 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


3-1 


Chapter  3:  Guidance  on  Important  SAM 
Processes  for  Initiating  an  Appraisal,  continued 


In  this  chapter  The  following  table  provides  a  guide  to  the  information  found  in  this 
chapter. 


Topic 

See  Page 

3.1  Using  Business  Goals  in  SAM 

3-3 

3.2  Tailoring  SAM  Based  on  Organizational  Context 

3-4 

3.3  Selecting  Appraisal  Personnel 

3-7 

3.4  Selecting  Projects  to  Be  Appraised 

3-9 

3.5  Using  the  Appraisal  Questionnaire 

3-11 

3.6  Developing  the  Rating  Profile 

3-15 

3.7  Developing  Findings 

3-24 
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3.1  Using  Business  Goals  in  SAM 


Assumptions 


Refining 
Appraisal  Focus 


The  following  discussion  presumes  that  an  organization  using  SAM  has 
a  strong  set  of  deployed  business  goals  at  its  disposal.  The  creation  and 
deployment  of  business  goals  is  a  task  of  significant  undertaking  and  is 
well  beyond  the  scope  of  this  method  description. 


An  organization  that  has  a  strong  sense  of  its  mission  and  business 
goals  can  optimize  the  use  of  SAM  by  analyzing  the  model  in  relation  to 
those  goals  and  deciding  on  a  focus  for  the  appraisal  that  may  be 
narrower  than  the  entire  model,  but  which  can  provide  more  direct  and 
focused  information  on  the  process  improvement  needs  of  the 
organization. 

Atematively,  a  broad  focus  may  still  be  desired  for  data  gathering; 
however,  the  focus  of  findings  and  recormnendations  can  be  limited  to 
those  which  directly  impact  the  business  goals.  This  provides  the 
breadth  of  data  gathering  that  allows  management  a  benchmark  of 
overall  capability  while  still  ensuring  a  focus  on  improvement  that  is 
directly  tied  to  the  business  needs  of  the  enterprise. 
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3.2  Tailoring  SAM  Based  on  Organizational  Context 


Many  factors 
influence 
organizational 
processes 


Organization 

structure 


Figure  3-1  illustrates  some  of  the  key  influences  organization's  must 
face  when  attempting  improvement  using  any  reference  model,  such  as 
the  SE-CMM.  The  discussion  following  Figure  3-1  is  intended  to  assist 
organizations  in  determining  some  of  the  tailoring  of  the  SAM  that  may 
be  necessary  to  support  improvement  in  their  particular  context. 


Organizational 


Guidance  by  SE-CMM 


■< 


Base 

Practice 


Generic 

Practice 


Figure  3-1.  Key  Influences  of  Organizational  Improvement. 


To  a  certain  extent,  SAM  has  been  designed  to  optimize  appraisal 
efficiency  in  an  organizational  setting  that  uses  projects  as  the  main 
means  of  accomplishing  work.  These  projects  are  assumed  to  be  led  by 
an  individual  or  team  who  have  resource  planning,  allocation,  and 
monitoring  responsibility,  as  well  as  responsibility  for  the  quality  and 
quantity  of  product  delivered.  Other  management  and  control  functions, 
such  as  staffing,  career  development,  and  administration,  may  or  may 
not  be  performed  within  the  project  boundaries. 

Common  organizational  structures  that  adapt  well  to  SAM  include  a 
project  structure  where  all  management,  staffing,  and  administrative 
functions  come  within  the  purview  of  the  project;  a  matrix  stmcture, 
where  the  project  planning,  resource,  monitoring  and  performance  are 
the  responsibility  of  the  project  with  staffing  and  other  administration 
being  handled  by  a  "functional"  or  "central"  organization;  and  integrated 
product  development  teams,  where  teams  are  created  from  functional  or 
product  line  groups  which  then  have  the  same  general  responsibilities  as 
a  project,  but  typically  for  a  smaller  scope  for  each  team. 

In  some  environments,  a  functional  structure,  where  the  resource 
planning  and  control  for  product  development  are  under  the  management 
of  a  central  or  functional  organization,  can  be  a  more  challenging 
environment  in  which  to  use  SAM  to  obtain  a  snapshot  of  organization- 
wide  systems  engineering  capability. 


continued  on  next  page 
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3.2  Tailoring  SAM  based  on  Organizational  Context,  continued 


Using  SAM  in 
functional 
organization 
structure 


SAM  assumes  that  there  is  someone,  in  SAM  called  the  Systems 
Engineering  Leader,  who  is  a  single  point  of  information  for  a  broad 
base  of  topics  related  to  the  systems  engineering  functions  being 
performed  in  projects  within  an  organization.  If  three  or  four  of  these 
SE  leads  are  interviewed  using  SE-CMM  as  the  reference  model,  the 
gaps  in  information  are  typically  relatively  small,  and  easily  filled  via  the 
practitioner  group  discussions. 

In  some  organizations  the  resource  control  is  held  by  a  functional 
manager,  either  related  to  life  cycle,  e.g.,  a  requirements  manager, 
design  manager,  integration  manager,  etc.;  or  by  product  line,  e.g.,  the 
product  line  A  manager,  product  line  B  manager,  etc.  The  "projects" 
constituted  to  support  this  type  of  organization's  needs  may  be  small 
enough  that  the  ongoing  overhead  of  a  project  management  stmcture  is 
inefficient,  or  this  structure  may  support  business  goals  of  the 
organization  in  some  way  other  structures  cannot. 

The  challenge  for  applying  SAM  is  in  trying  to  find  a  single  interview 
source  for  the  broad  information  gathering  needed  at  the  beginning  of 
SAM.  This  may  not  be  possible.  Where  infeasible,  tailoring  the 
method  to  achieve  the  goals  of  getting  both  breadth  of  information 
typically  provided  by  the  SE  leads,  and  depth  of  information  typically 
provided  by  the  practitioner  groups,  is  the  likely  alternative.  TWs  may 
entail  changing  the  on-site  schedule  for  the  appraisal,  re-defining  the 
"organization"  being  appraised,  distributing  the  questionnaire  across  a 
broader  range  of  projects  and  then  selecting  interviewees  based  on  the 
breadth  of  Imowledge  exhibited  by  their  questionnaire  responses,  or 
other  alternatives. 


continued  on  next  page 
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3.2  Tailoring  SAM  based  on  Organizational  Context,  continued 


Using 

organizational 

development 

resources 


Organizational  developnient(OD)  is  a  discipline  within  the  social  and 
decision  sciences  which  attempts  to  understand  the  behavior  of  large  and 
small  workgroups,  as  well  as  design  and  execute  interventions  needed 
to  optimize  organizational  functioning,  enable  needed  changes  to  occur, 
and/or  positively  influence  aberrant  organizational  behaviors.  Many 
product  development  enterprises  have  OD  staff,  either  in  the  training 
department,  or  in  a  specific  organization  called  Organizational 
Development  or  something  similar.  OD  specialists  are  frequently  a 
valuable  source  in  determining  how  to  tailor  SAM  for  a  particular 
organizational  context,  especially  one  that  does  not  immediately  appear 
to  follow  one  of  the  patterns  expected  by  SAM.  They  can  also  be 
helpful  in  selecting  appraisal  team  members,  appraisal  participants, 
projects,  and  improvement  strategies,  as  well  as  help  to  plan  and 
implement  the  organizational  improvement  plan. 

OD  staff  may  be  particularly  helpful  in  determining  the  most  appropriate 
project  lead/functional  management  mix  for  functional  style 
organizations. 


Product  line  Many  organizations  today  are  using  product  lines  as  a  method  of 

focus  defining  organizational  boundaries.  When  applying  SAM  in  such  an 

organization,  an  appraisal  focusing  within  a  single  product  line  is  likely 
to  produce  the  most  actionable  results.  Crossing  product  lines  can  give 
a  perspective  on  the  overall  process  strengths  and  weaknesses  of  the 
organization,  but  is  less  likely  to  come  up  with  focused  findings  that  can 
be  leveraged  from  for  significant,  timely  improvement. 
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3.3  Selecting  Appraisal  Personnel 


Guidance  for  The  facilitators  and  appraisal  team  members  drawn  from  the 
selecting  the  organization  work  as  a  single  team  during  the  entire  on-site  phase.  The 

appraisal  team  appraisal  team  will  analyze  data,  perform  all  of  the  interviews,  and 

develop  the  findings.  They  are  also  responsible  for  developing  the 
recommendations  to  the  findings  which  are  published  in  the  appraisal 
report,  and  should  be  involved  in  carrying  process  improvement 
activities  throughout  the  organization.  To  assure  transition  from  the 
appraisal  to  a  robust  improvement  effort,  the  appraisal  team  members 
should 

•  Be  advocates  of  process  improvement. 

•  Be  credible  with  both  management  and  participants. 

•  Be  involved  in  action  planning  and  the  subsequent  improvement 
effort. 

•  Have  good  communication  skills. 

•  Have  a  positive  and  encouraging  attitude. 

If  the  organization  has  any  previous  experience  with  organizational 
appraisals  and  ensuing  process  improvement  activities,  such  as 
improvement  based  on  the  CMM  for  Software,  it  can  be  advantageous 
to  include  a  person  who  has  been  involved  in  those  activities.  Such  a 
person  brings  appraisal  experience  from  within  the  organizational 
culture,  and  can  be  used  as  a  source  to  gain  leverage  from  process 
improvement  activities  that  have  worked  within  the  organizational 
culture.  These  individuals  are  also  likely  candidates  to  be  SAM 
facilitators. 

continued  on  next  page 
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3.3  Selecting  Appraisal  Personnel,  Continued 


Why  use  systems 
engineering  leads 
and  practitioners 
as  primary  data 
sources? 


The  SE-CMM  appraisal  method  is  structured  to  examine  two  views  of  the 
entity  being  appraised,  typically  project  and  organizational  support.  The 
objective  is  to  get  as  broad  a  view  of  the  organization  as  possible  while 
still  maintaining  control  of  the  data  gathering  process.  The  project  lead 
systems  engineers  are  typically  the  only  ones  who  complete  the 
questionnaire  since  they  have  a  broader  view  of  the  tasks  being  performed 
in  systems  engineering  and  associated  support  functions  than  most  other 
roles  in  a  product  development.  The  questionnaires  are  essential  to 
establishing  the  basis  for  data  gathering  for  the  entire  on-site  week.  The 
practitioners  expand  the  appraisers'  view  of  the  organization,  providing 
information  on  what  is  being  done  on  other  projects  or  areas  of  the 
organization  besides  those  targeted  for  in-depth  data  gathering.  The 
appraisal  team  may  choose  to  have  select  people  in  the  practitioner  group 
complete  the  questionnaire,  but  this  is  typically  kept  to  a  limited  number 
for  the  purpose  of  data  and  resource  management. 


On  some  appraisals  the  appraised  organization  may  be  a  large  project. 
In  that  situation  the  "projects"  may  be  large  subsystems  or  segments  of 
the  project.  The  project  lead  systems  engineers  would  then  be  the 
segment  lead  systems  engineers,  and  the  practitioners  would  be  drawn 
from  other  project  segments. 
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3.4  Selecting  Projects  to  Be  Appraised 


Project  selection 
considerations 


Project  location 


The  types  of  projects  selected  to  participate  in  the  appraisal  are  based  on 
the  goals  of  the  appraisal.  Table  3-1  lists  some  considerations  in 
selecting  projects. 


Goal  of  the  Appraisal 

Type  of  Projects  to  Select 

Understand  domain-related 
issues. 

Select  projects  within  desired  domain. 

Note  that  the  domain  can  be  focused  on 
any  one  of  the  following  factors:  industry, 
technology  base,  customer  type,  project 
complexity,  etc. 

Understand  deployment  of 
new  organization^ 
practices. 

Select  new  projects  that  have  started  since 
the  deployment  of  new  practices. 

Determine  overall  capability 
of  the  organization. 

Select  projects  that  are  expected  to  be 
representative  of  the  organizational 
capability. 

Determine  progress  of 
process  improvement 
activities. 

Select  projects  that  have  been  the  pilots 
for  process  improvements. 

Table  3-1.  Project  Selection  Considerations. 


Although  it  is  not  required  by  SAM  that  the  projects  of  an  appraisal  be 
near  each  other  geographically,  the  logistics  considerations  associated 
with  a  multi-site  appraisal  should  be  carefully  considered.  The  appraisal 
interview  and  feedback  process  assumes  that  the  interviewers  will  be 
available  for  multiple  sessions  during  the  on-site  week.  Tailoring  to 
accommodate  multi-site  appraisals  should  account  for  the  need  for  the 
interview/feedback  loop  if  activities  are  considered  for  resequencing. 
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3.4  Selecting  Projects  to  be  Appraised,  Continued 


Guidelines  for 
selecting 
practitioner 
groups 


There  are  two  issues  associated  with  the  selection  of  the  practitioner 
groups: 

•  Ensuring  appropriate  functional  areas  are  represented. 

•  Ensuring  the  right  type  of  person  is  participating  in  the  group. 

The  practitioner  groups  should  represent  the  primary  systems 
engineering-related  tasks  performed  within  the  organization  (e.g., 
analysis,  requirements,  test).  However,  it  is  important  to  include  the 
support  organizations  (specifically  quality,  configuration  management, 
and  training)  even  if  these  functions  are  not  encompassed  structure- 
wise,  in  the  organization  being  appraised.  It  is  also  recommended  that 
practitioners  from  specialty  disciplines  who  support  systems 
engineering  (e.g.,  human  factors,  reliability,  manufacturing)  be  included 
in  the  practitioner  sample. 

The  people  who  are  selected  to  be  in  the  practitioner  groups  should 

•  Be  opinion  leaders  who  are  credible  with  their  peer  group. 

•  Be  drawn  from  areas  widespread  throughout  the  organization. 

•  Be  willing  to  communicate  and  express  their  opinions. 

•  Be  able  to  talk  freely  and  supply  useful  information. 

•  Not  suppress  or  intimidate  any  conversation. 

To  ensure  candid  conversation  in  the  practitioner  groups,  it  is  typically 
not  recommended  to  have  line  management  represented.  If  one  of  the 
goals  of  the  appraisal  is  to  assess  management  issues,  the 
recommendation  is  to  have  an  entire  practitioner  group  composed  of 
managers. 
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3.5  Using  the  Appraisal  Questionnaire 


Questionnaire 

distribution 


Recommended 

questionnaire 

recipients 


The  questionnaire  is,  at  a  minimum,  distributed  to  the  systems 
engineering  leads  or  equivalent.  However,  depending  upon  the 
responsibilities  and  visibility  of  the  systems  engineering  leads  within  the 
organization,  the  questionnaires  for  some  of  the  process  areas  may  be 
distributed  to  people  who  are  more  familiar  with  the  area  covered  by  the 
process  area  than  the  systems  engineering  lead.  Specifically,  the 
organizational  process  areas  should  be  completed  by  the  appropriate 
subject  matter  experts. 


To  maximize  the  accuracy  of  initial  responses  to  the  questions  in  the  SE- 
CMM  questionnaire,  it  is  recommended  that  the  questionnaires  be 
distributed  to  individuals  with  the  skills  and  roles  expressed  in  Table  3-2. 


Process  Area 

Primary  Recipients 

Secondary 

Recipients 

01:  Analyze  Candidate 
Solutions 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Any  senior  practitioner 
with  significant  system 
design  experience 

02:  Derive  and 

Allocate  Requirements 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

03:  Develop  Physical 
Architecture 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

04:  Integrate 
Disciplines 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Senior  specialty 
engineers  (e.g., 
reliability,  safety, 
manufacturing,  human 
factors)  working  on  the 
projects  selected  for 
appraisal 

05:  Integrate  System 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

06:  Understand 
Customer  Needs  and 
Expectations 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

•  Technical  marketing 
personnel 

•  Proposal  personnel 

•  Customer  service 
personnel 

Table  3-2.  Questionnaire  Distribution  Table. 
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3.5  Using  the  Appraisal  Questionnaire,  continued 


Recommended 

questionnaire 

recipients, 

continued 


Process  Area 

Primary  Recipients 

Secondary 

Recipients 

07:  Verify  and 

Validate  System 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

System  verification 
manager  or  senior  test 
engineers 

08:  Ensure  Quality 

Senior  project-level 
quality  manager  or  lead 
(in  environments  with 
shared  quality 
leadership 

responsibility,  systems 
engineering  lead  for  the 
project) 

•  Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

•  Organization^ 
quality  manager,  total 
quality  management 
coordinator 

09:  Manage 
Configurations 

Senior  project-level 

CM  manager  for  the 
projects  selected  for 
appraisal 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

10:  Monitor  and 
Control  Technical 

Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

11:  Plan  Technical 
Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

12:  Manage  Risk 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Project  or  program 
manager  for  the 
projects  selected  for 
appraisal 

13:  Define 
Organization's 

Systems  Engineering 
Process 

Individuals  responsible 
for  defining 
organization-level 
processes;  may  be  part 
of  the  quality 
leadership  area, 
policies/procedures 
area,  or  other  support 

group _ 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Table  3-2.  Questionnaire  Distribution  Table,  continued 
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3.5  Using  the  Appraisal  Questionnaire,  continued 


Recommended 

questionnaire 

recipients, 

continued 


Process  Area 

Primary  Recipients 

Secondary 

Recipients 

14:  Improve 
Organization's 

Systems  Engineering 
Processes 

Individuals  responsible 
for  deploying 
organization-level 
process  improvement 
activities;  may  be  part 
of  the  quality 
leadership  area, 
policies/procedures 
area,  or  other  support 
group 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

15:  Manage  Product 
Line  Evolution 

Individuals  at 
organization  level 
responsible  for 
strategic  product  line 
positioning  and 
advancement;  may  be 
in  R&D,  technical 
marketing,  or  other 
support  structure 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

16:  Manage  Systems 
Engineering  Support 
Environment 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Individuals  at 
organization  level 
involved  in  deploying 
new  development 
technologies 

17:  Manage  Systems 
Engineering  Training 

Individuals  responsible 
for  organization-level 
training  planning, 
development,  and 
deployment;  may  be 
part  of  an  R&D  group, 
training  department,  or 
other  support  structure 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Table  3-2.  Questionnaire  Distribution  Table,  continued 
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3.5  Using  the  Appraisal  Questionnaire,  continued 


Questionnaire 

administration 


The  questionnaire  must  be  completed  prior  to  the  on-site  phase.  We 
recommend  that  it  be  administered  with  a  facilitator  or  site  coordinator 
present  and  administered  to  the  systems  engineering  leads  and  other 
appropriate  personnel  as  a  group,  so  that  common  questions  can  be 
answered.  The  typical  question  regards  organizational  roles.  The 
person  administering  the  questionnaire  should  emphasize  that  the  SE- 
CMM  is  role  independent.  The  questionnaire  only  asks  if  specific  tasks 
are  performed  on  a  project,  not  if  they  are  performed  by  a  specific 
person. 
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3.6  Developing  the  Rating  Profile 


Introduction 


Diagram  of 
rating  proHle 
development 
process 


One  of  the  results  of  the  on-site  phase  is  a  rating  profile  covering  the 
appraised  process  areas.  The  rating  profile  correlates  closely  with  the 
appraisal  findings,  and  the  two  are  developed  in  a  closely  coupled 
process. 

The  rating  profile  is  developed  and  refined  at  specific  points  in  the 
appraisal  process,  as  detailed  in  the  process  element  summaries  in 
Chapter  2.  At  the  end  of  the  primary  data-gathering  activities,  prior  to 
generating  preliminary  findings,  the  team  reviews  its  data  tracking 
sheets  (DTSs)  to  formulate  a  preliminary  rating  for  each  process  area. 
The  DTS  is  a  support  tool  introduced  in  Appendix  C  to  help  organize  the 
data  obtained  by  the  appraisal  team.  Ratings  are  based  on  the  degree  to 
which  the  appraised  entity  performs  all  of  die  practices  at  a  given  level, 
in  the  judgment  of  the  appraisal  team.  Issues  raised  in  generating 
preliminary  ratings  can  be  addressed  during  the  preliminary  feedback 
sessions.  The  rating  is  then  refined  into  a  draft  rating  in  conjunction 
with  developing  the  draft  findings.  Prior  to  drafting  final  findings,  the 
team  once  again  reviews  the  DTSs  to  formulate  the  final  rating  for  each 
of  the  process  areas. 


Sources  of _ Individual  _  Team  _  Team 

Information  Evaluation  Consensus  Outputs 


Iterate 


Figure  3-2.  Rating  Profile  Development  Process. 
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3.6  Developing  the  Rating  Profile,  continued 


Data  tracking 
sheet 


The  DTS  is  the  primary  data  management  tool  used  by  the  appraisal 
team  during  the  on-site  phase.  The  responses  to  the  questionnaires  are 
transcribed  onto  the  DTS  at  the  beginning  of  the  on-site  phase.  Trends 
or  inconsistencies  in  data  can  be  identified  more  easily  in  the  DTS 
format,  which  forms  the  basis  for  exploratory  questions  for  the  systems 
engineering  lead.  Information  obtained  from  the  systems  engineering 
lead  and  practitioner  interviews  that  supports  (corroborating  data)  or 
contradicts  (opposing  data)  the  premise  that  the  base  or  generic  practice 
characteristic  is  exhibited,  is  then  recorded  on  the  DTS.  All  information 
recorded  on  the  DTS  is  used  to  develop  the  rating  profile,  in  addition  to 
the  appraisal  team's  notes  from  interviews  and  reviews  of  supporting 
material. 

The  DTS  format  also  facilitates  identifying  where  no  additional 
supporting  data  have  been  obtained.  This  permits  directing  further 
interviews,  particularly  the  practitioner  interviews,  to  fill  in  any 
information  gaps.  Conscientious  data  management  throughout  the  on¬ 
site  phase  significantly  improves  the  efficiency  of  the  whole  appraisal 
process  and  facilitates  the  rating  development. 
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3.6  Developing  the  Rating  Profile,  Continued 


DTS  Figure  The  following  is  a  condensed  version  of  the  data  tracking  sheet  found  in 

Appendix  C. 


Questionnaires 

Initial 

A 

B 

C 

D 

E 

F 

G 

Projected 

Rating 

PA  01:  Analyze  Candidate  Solutions 

Base  Practices: 

I.l 

Establish  Evaluation  Criteria 

1.2 

Define  Approach 

1.3 

Identify  Additional  Alternatives 

1.4 

Analyze  Candidate  Solutions 

1.5 

Select  Solutions 

1.6 

Capture  Results 

z 

Generic  Practices: 

2.1.1 

Allocate  Resources 

2.1.2 

Assign  Responsibilities 

2.1.3 

Document  Approach 

2.1.4 

Provide  Tools 

2.1.5 

Ensure  Training 

2.1.6 

Plan  Performance 

2.2.1 

Follow  Plans 

2.2.2 

Do  Configuration  Management 

2.3.1 

Verify  Process  Compliance 

2.3.2 

Audit  Work  Products 

2.4.1 

Track  with  Measurement 

2.4.2 

Take  Corrective  Action 

3.1.1 

Standardize  Process 

3.1.2 

Tailor  Standard  Process 

3.2.1 

Follow  Defined  Process 

3.2.2 

Perform  Peer  Reviews 

3.2.3 

Use  Process  Data 

4.1.1 

Establish  Quality  Goals 

4.2.1 

Determine  Process  Capability 

4.2.2 

Use  Process  Capability 

5.1.1 

Establish  Process  Goals 

5.1.2 

Improve  Defined  Process 

1  5.2.1 

Do  Causal  Analysis 

5.2.2 

Eliminate  Defect  Causes 

5.2.3 

Improve  Standard  Process 

Figure  3-3.  Condensed  Data  Tracking  Sheet. 
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3.6  Developing  the  Rating  Profile,  continued 


DTS  Figure, 
continued 


Interviews 

Prelim 

A 

B 

C 

D 

Gp  1 

Gp  2 

Gp  3 

Rating 

PA  01:  Analyze  Candidate  Solutions 

J. 

Base  Practices: 

1.1 

Establish  Evaluation  Criteria 

1.2 

Define  Approach 

1.3 

Identify  Additional  Alternatives 

1.4 

Analyze  Candidate  Solutions 

1.5 

Select  Solutions 

1.6 

Capture  Results 

n 

Generic  Practices: 

<sanie  as  above> 

Findings  Review 

Final 

A 

B 

C 

D 

0th 

Oth 

Oth 

Rating 

PA  01:  Analyze  Candidate  Solutions 

J 

Base  Practices: 

1.1 

Establish  Evaluation  Criteria 

1.2 

Define  Approach 

1.3 

Identify  Additional  Alternatives 

1.4 

Analyze  Candidate  Solutions 

! 

1.5 

Select  Solutions 

1.6 

Capture  Results 

z 

Generic  Practices: 

<same  as  above> 

Figure  3-3.  Condensed  Data  Tracking  Sheet,  continued 
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3.6  Developing  the  Rating  Profile,  Continued 


Preliminary 

ratings 


Draft  and  Onal 
ratings 


Using  rating  data 


No  attempt  should  be  made  to  develop  a  preliminary  rating  until  all  of 
the  primary  data  have  been  collected  (all  initial  systems  engineering  lead 
and  practitioners  have  been  interviewed,  and  supporting  material  has 
been  reviewed).  Preliminary  ratings  are  developed  at  the  same  time  as 
the  preliminary  findings,  typically  during  the  afternoon  of  the  third  on¬ 
site  appraisal  day.  The  advantage  of  developing  preliminary  ratings  at 
this  time  is  that  the  need  for  additional  information  may  be  brought  up 
while  developing  the  ratings.  There  is  a  further  opportunity  to  get 
additional  information  during  review  sessions  of  the  preliminary 
findings  with  the  systems  engineering  leads. 

Preliminary  ratings  are  a  synthesis  of  all  of  the  primary  data  collected. 
The  appraisal  team  must  determine  if  the  base  or  generic  practice  is 
performed  based  upon  the  data  shown  on  the  DTS.  Conflicting  data 
must  be  resolved  by  appraisal  team  consensus. 


The  final  ratings  are  developed  prior  to  developing  the  final  findings. 
They  serve  as  a  starting  point  for  developing  the  final  findings.  There  is 
generally  a  close  correlation  between  the  final  findings  and  the  process 
areas  with  a  low  rating  profile. 

Information  from  the  review  sessions  of  the  preliminary  and  final 
findings  is  used  to  adjust  the  ratings  as  needed. 


The  rating  algorithm  for  determining  a  capability  level  in  a  process  area 
is  the  appraised  entity  performing  100%  of  the  base  or  generic  practices 
in  the  respective  level.  In  addition,  higher  level  can  be  achieved  only  if 
a  lower  level  is  also  fulfilled. 

Performance  of  100%  of  the  base  or  generic  practices  is  estimated  for 
the  appraised  entity  as  a  whole  only  at  the  preliminary,  draft,  and  final 
ratings  steps.  As  explained  in  further  detail  in  the  DTS  Summary  in 
Appendix  C,  preliminary,  draft,  and  final  findings  and  the  rating  profile 
represent  a  summary  of  all  of  the  data  collected  over  the  appraisal  on-site 
period. 
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3.6  Developing  the  Rating  Profile,  continued 


Possible  rating 

presentation 

formats 


Example  pie 
chart 


The  following  formats  for  presenting  the  rating  profile  have  all  been 
used  in  different  appraisal  contexts. 

•  Pie  chart:  number  of  PAs  at  specific  levels  (works  well  for 
organization  primarily  in  the  level  2-3  maturity  range). 

•  Bar  chart:  score  for  each  PA  shown  on  a  bar  graph  which  allows  the 
option  to  use  quantiles  for  PAs  below  level  1  (good  option  for  the 
organization  in  the  0-2  maturity  range). 

•  Tabular  format. 

Other  possible  presentation  formats  include  the  Kiviat  diagram  (also 
called  a  radar  chart  or  bull's  eye  chart). 


Profile-Pie  Chart 

PAs  at  level  2 


Figure  3-4.  Pie  Chart  Example. 
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3.6  Developing  the  Rating  Profile,  Continued 


Example  bar 
chart 


Figure  3-5.  Bar  Chart  Example. 
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3.6  Developing  the  Rating  Profile,  continued 


Example  tabular 
format 


PA  Title 

Rating 

Analyze  Candidate  Solutions 

3 

Develop  Functional/Performance  Requirements 

1 

iDevelop  Physical  Architecture 

2 

Integrate  Disciplines 

0 

Integrate  System 

3 

Understand  Customer  Needs  &  Expectations 

1 

Verify  &  Validate  System 

2 

Ensure  Quality 

1 

Manage  Configurations 

1 

Manage  Risk 

0 

Monitor  &  Control  Technical  Effort 

1 

Plan  Technical  Effort 

1 

Define  Organization's  Systems  Engineering  Process 

0 

Improve  Organization's  Systems  Engineering  Processes 

0 

Manage  Product  Line  Evolution 

1 

Manage  Systems  Engineering  Support  Environment 

1 

Manage  Systems  Engineering  Training 

0 

Table  3-3.  Tabular  Format  Example. 
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3.6  Developing  the  Rating  Profiie,  continued 


Example  Kiviat 
diagram 


Profile-Kiviat  Diagram 


Figure  3-6.  Kiviat  Diagram  Example. 
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3.7  Developing  Findings 


Introduction 


Exploratory 

questions 


The  appraisal  findings  are  a  key  product  of  the  appraisal.  They  are  the 
result  of  synthesizing  all  of  the  data  collected  throughout  the  on-site 
phase,  along  with  the  questionnaire  responses.  Findings  are  limited  to 
approximately  seven,  so  that  the  organization  is  left  with  a  number  of 
findings  that  is  manageable  in  forming  or  continuing  an  improvement 
effort. 


Exploratory  questions  are  developed  to  explore  the  reason  for  any 
inconsistencies  in  the  questionnaire  responses,  and  are  the  first  step  in 
eliciting  supporting  or  conflicting  information  on  the  performance  of 
base  or  generic  practices.  Exploratory  questions 

•  Should  be  limited  to  approximately  50  per  systems  engineering  lead, 
and  should  be  prioritized  due  to  interview  time  limitations. 

•  Should  be  linked  to  the  individual  process  areas  in  order  to  maintain 
traceability  to  the  model,  and  facilitate  the  data  management  that  needs 
to  occur  tWughout  the  on-site  phase. 

•  Are  typically  a  mix  of  some  specific  questions  that  are  designed  to 
address  inconsistencies  on  a  specific  project,  or  general  questions 
aimed  at  possible  consistent  misinterpretations  of  the  questionnaire. 

Information  on  the  performance  of  base  or  generic  practices  is  recorded 
on  the  DTS  after  each  of  the  interviews  with  the  systems  engineering 
lead. 
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3.7  Developing  Findings,  Continued 


Preliminary 

findings 


Process  for 
developing  draft 
findings 


The  preliminary  findings  are  a  synthesis  of  the  primary  issues  that 
appraisal  team  members  have  collected  from  both  the  systems 
engineering  leads  and  the  practitioner  groups.  Preliminary  findings 
include  both  strengths  and  weaknesses. 

The  preliminary  findings  are  reviewed  with  the  systems  engineering 
leads  to  confirm  that  the  findings  are  true  for  their  project,  and  if  the 
findings  are  true  for  the  organization  based  upon  their  knowledge.  The 
primary  objective  of  this  set  of  sessions  to  validate  the  synthesized 
comments. 

Feedback  from  the  systems  engineering  leads  also  helps  in  the 
prioritization  of  the  data.  The  number  of  times  that  a  preliminary  finding 
is  supported  or  contradicted  helps  the  appraisal  team  determine  fiigh- 
priority  items  in  the  next  synthesis  step  to  develop  the  draft  findings. 

An  additional  objective  of  the  feedback  sessions  is  to  obtain  more 
supporting  or  conflicting  information  that  is  used  to  establish  the  rating. 
Additional  questions  may  be  inserted  to  address  uncertainties  that  the 
appraisal  team  may  have  with  respect  to  the  rating  profile. 


The  draft  findings  are  the  unreviewed  version  of  the  final  findings. 
Confirmed  preliminary  findings  are  clustered  using  a  technique  such  as 
affinity  diagram,  and  a  set  of  5  to  10  themes  or  underlying  factors  are 
derived  which  forms  the  draft  findings.  Draft  findings  are  presented  via 
briefing  charts  in  the  format  of  finding,  cause,  and  consequence. 

The  draft  and  final  findings  typically  fall  into  one  of  three  categories: 

•  General  barriers  to  the  next  level. 

•  Weaknesses  in  the  base  practices. 

•  Weaknesses  in  the  generic  practices. 

The  draft  findings  are  presented  to  the  systems  engineering  leads  and 
practitioners  to  get  validation  of  the  draft  findings,  give  the  appraisal 
team  leader  a  chance  to  dry-run  the  presentations,  and  allow  a  forum  for 
the  practitioners  to  refine  the  findings. 


continued  on  next  page 
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3.7  Developing  Findings,  continued 


Finding,  cause, 
consequence 


Final  findings 


Once  a  set  of  findings  areas  are  identified,  the  team  should  brainstorm 
the  findings,  causes,  and  consequences.  The  main  findings  should  be 
carefully  worded  to  reflect  what  the  team  has  actually  heard  from  the 
participants.  The  finding  is  usually  a  single  observation;  it  may  be 
thought  of  as  a  characterization  of  a  symptom.  An  example  of  a  finding 
is 

System  engineering  plans  do  not  realistically  reflect  the  needs  of  the 
projects. 

Causes  are  observations  that  support  the  central  finding,  and  many 
indicate  potential  causes  of  the  finding.  An  example  of  a  cause  is 

Estimates  are  not  based  on  available  historical  data. 

Consequences  list  the  probable  results  of  the  finding.  An  example  of  a 
consequence  is 

Cost  &  schedule  overruns 

The  causes  and  consequences  are  often  taken  directly  from  the 
preliminary  findings.  The  causes  are  an  opportunity  to  employ  the 
phrasing  heard  during  the  practitioner's  sessions.  On  the  other  hand, 
consequences  primarily  target  the  sponsor  and  should  reflect  his/her 
perspective.  Use  of  organizational  objectives  or  goals  and  the  risks  to 
meeting  those  goals  is  recommended  as  part  of  the  consequences. 

During  this  initial  step  do  not  get  bogged  down.  If  the  team  cannot 
agree  on  wording,  leave  it  for  the  next  step.  At  the  end  of  this  step,  the 
findings  have  been  identified  along  with  a  set  of  causes  and 
consequences. 


Final  findings  are  an  edited  version  of  the  draft  findings  based  upon 
comments  from  the  practitioners  and  systems  engineering  leads.  These 
are  used  to  develop  the  appraisal  report  and  recommendations. 

A  sample  final  findings  briefing  is  included  in  Appendix  B  for 
reference. 
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Appendices 


Introduction 


In  these 
appendices 


These  appendices  provide  support  materials  that  organizations  using 
SAM  may  find  helpful  in  constructing  appraisal  artifacts  and  training 
team  members. 
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Appendix  A:  Template  for  SAM  Opening  Briefing 


Introduction 


t 


The  opening  briefing  template  provides  ideas  on  how  to  present  the 
basic  concepts  of  SAM  and  the  SE-CMM  to  the  appraisal  participants. 
However,  appraisers  are  welcome  to  design  whatever  materials  suit  their 
style/needs.  The  basic  purpose  of  the  opening  briefing  and  suggested 
contents  are  found  in  Section  2.2.1. 

Also  included  in  this  appendix  are  some  of  the  basic  information  charts 
included  in  the  SE-CMM  Project  Overview  briefing.  For  audiences 
who  are  somewhat  unfamiliar  with  the  model,  these  charts  or  similar 
ones  may  be  of  use  as  backup  charts  for  the  opening  briefing  to  help 
answer  questions  about  the  model  itself. 


continued  on  next  page 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  Continued 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


Process  Maturity  Models 

A  maturity  model  defines  the  requirements  for  a  process 
-Defines  “what”  NOT  “how” 

—Does  NOT  address  People  &  Technology 
Framework  for  describing  key  elements  of  an  effective  process 
—  Requirements  for  Process  Definition 
-  Guidance  for  Process  Improvement 
Yardstick  for  judging  the  maturity  of  an  organization’s  process 
-Contractor  Selection 

—Process  Improvement  Metric _ 


f  \ 

Process  Maturity  Elements 

•  Process  Elements 

“What you  dd’ 

—  Process  Areas 

-  Base  Practices 

*  Maturity  Elements 

“How  well  you  do  it’ 

—  Maturity  Levels 

—  Common  Features 

—  Generic  Practices 

•  Assessment  Process 

“How  we  measure  if' 

.j 

continued  on  next  page 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


Process  Improvement 


“If  you  don’t  know  where  you  are,  a  map  won’t  help.” 


Maturity  Model 
Assessment  Findings 
Recommendations 


Action  Plan 


discover  where  you  are 
decide  where  you  want  to  be 
plan  how  to  get  there 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  Continued 


Confidentialit 


The  assessment  depends  on  your  frank  &  open  discussions! 

•  No  project  or  individual  will  be  identified  in  the  findings 

•  The  team  will  not  discuss  your  comments  outside  the 
assessment 

•  We  expect  you  not  to  discuss  what  you  hear  during 
our  meetings 


Tight  schedule  -  Meetings  will  start  on  time! 


Assessment  -  Results 


Products 

—  Findings  Briefing 

—  Level  of  maturity  for  each  process  area 
Next  Steps 

—  Final  Briefing 

—  Findings  &  Recommendations  Report 

—  Action  Plan 


SECMM-94-06ICMU/SEI-94-HB-05  v1.0 


Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


Extra  briefing 
charts 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


SE-CMM _ 

Industry  —  Academia  Government 


^Systems  Engineering  improvement 


f  Define 
rganization's 
SE  Process 


Improve 
Organization's 
>E  Processes 


nonhor/Control 
1  Effort 


I  Derive  & 
Allocate 
equirements 


Manage  SE 
Support 
Environment 


Manage  SE 
Training 


I  Manage 
Configurations 


Manage  ^ 

Evolution  1 

1  Quality 

»  «  collatwraliv*  offort  of  GTE  Gov^mmont  SyttBfn*.  Hugh#*  Airaaft  Corjw«l(On,  Lockh**<J  Conwalor.  Loral  Corporation.  Iho  SoBwaro 
Prodoatvity  Consortium.  Toxas  Inalrumonls,  Ineorporalad,  and  tho  SoBwaro  Enginaanng  InsIHuta.  Parmaiion  to  copy  if  granlad  provKjoeniua^ica  appoars 


SE-CAfM 

Industry  —  At 


Academia  — -  Government 


Process  Area  List 


^Systems  Engineering  Improvement 


Engineering 


Understand  customer 
needs  and  expectations 

Derive  and  allocate 
requirements 

Analyze  candidate 
solutions 

Develop  physical 
architecture 

Integrate  system 
Verify  &  validate  system 
Integrate  disciplines 


Project 


Manage 

configurations 

Plan  technical  effort 

Monitor  and  control 
technical  effort 

Manage  risk 
Ensure  quality 


Organization 


Define  organization’s 
systems  engineering 
process 

Manage  systems 
engineering  support 
environment 
Manage  systems 
engineering  training 

Manage  product  line 
evolution 

Improve  organization’s 
systems  engineering 
processes 


Thi*  work  Is  a  colLaboraItvo  alTort  of  GTE  Govammonl  Systams,  Hughas  ArcraB  Corporation,  Locfcheod  Corporation.  Loral  Corporation  tha  SoBwars 
ProdudMiy  Coiwoftium.  Toxas  Instrumants.  Ineorporatod.  and  Iha  SoBwaro  Engrnoanng  Inslitulo.  Parrmtion  to  copy  is  granlod  providod  thi?nolic«  appoars. 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  Continued 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  Continued 


SE-CMM 

rauslry  --  Academia  --  <:iovernmen. - ^Systems  Engineering  Improvemen, 

‘T3^ical”  Process  Area 


PA  05:  Integrate  System 

•  The  purpose  of  Integrate  System  Is  to  ensure 
that  system  elements  will  function  as  a 
whole.  This  primarily  involves  identifying, 
defining,  and  controlling  interfaces,  as  well 
as  verifying  system  functions  that  require 
multiple  system  elements. 


Thi»  woric^ttcollabofatrv«*frort  of  GTE  Government  Syitome,  Hoghe*  Arcraft  Corporation,  Lockhood  Corporation,  Loral  Corporation,  the  Software 
Productrvity  Consortium,  Texas  Jnsiruments.  fncorporaied.  and  the  Software  Engirwering  Institute.  Permisiori  to  copy  is  granted  provided  ihi*  notice  appears 


SE-CMM 

industry  Academia  -  Government  ^  Syelews  Engineering  Improvement 


'Typical  Generic  Practice” 


Generic  Practices 

1 1 .0  Performed  Level 

Common  Feature  1:  Base  Practices  are 
performed 

1.1.1  Perform  the  process.  Perform  a  process 
that  implements  the  base  practices  of  the 
process  area  to  provide  work  products  and/or 
services  to  a  customer. 


^MU.  199^  This  work  it  a  collaborative  effort  of  GTE  Goverrment  Sysiems.  Hughes  Aircraft  Corporaiior>,  Lockheed  Cotporaiion,  Loral  Corporation  the  Software 
Producttvity  Consortwm,  Texas  Insirumenls,  (neorperalad.  and  Ihe  Software  Engjneenng  IntUtufe,  Permission  to  copy  is  vantod  provided  this  notice  aopears 


continued  on  next  page 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 


SE-CMM 

Industry  —  Ac 


Academia  Government 


Relationships 


^Systems  Engineering  Improvement 


Capability  Level  2 
I  Planned  &  Tracked 


Common  Features 


lanning  Performance 
disciplined  Performance 
racking  Performance 
erifying  Performance 


Generic 

Practices 


Planned  &  Tracked 
LI  Planning  Perfonnanc* 

L1.1  Allocate  resources 

2.1.2  Assign  resportsibilities 

2.1.3  Document  the  process 

2.1.4  Provide  tools 

2.1.5  Ensure  training 

2.1 .6  Plan  the  process 


TOW.  1  Thi*  work  w  a  collabofaljva  atTort  of  GTE  Gwammanl  Syalame.  Hughaa  Alreraft  Cofporalion.  Lockhaad  Corporation,  LoraJ  Corporation.  Iho  Sotlwara 
ProCuetrvrly  Contortium.  Taxaa  Inatnjtnanl*.  Ineorporatad.  and  tha  Sotlwara  Enginaaring  Inatitula.  Parmiaaton  to  copy  ia  grantad  providad  thia  notica  appaaia. 


SE  ■  CMM 


Systems  Engineering  Improvement 


Roadmap  for  Improvement 


NU I  I  Baso  practr 

PERFORMED  Ipartormed 


■  •  Committing  to 
portorm 

•  Planning 
parformance 

•  Olsciplinod 
performanco 

•Tracking 

performance 

•  Verifying 
performance 


WetL'Perffilgg^  Establishing 

measurable  quality 
►Definir^a  goals 

Standard  process  “Determining 

process  capability  to 
standard  process  _ Z,. 


standard  process 

•  Using  data 

•  Perform  the 
defined  process 


achieve  goals 
Objectively 
managing 
performance 


improving  process 
bffectivaness 


Common 

Features 


Thr*  wrk  i*  «  cottabomW®  affoft  o\  GTE  Gov«mnf»nt  Syiiomt,  Kughe«  Aircrafl  Corporation.  Lookho«f  Corporation.  Loral  Corporation,  tho  Soflwaro 
PfO^jctivrty  Consortium.  Twas  Instrurnonta,  Incorporaiod.  and  Iho  Scftwaro  Engirwaring  Instituio.  Parnission  to  copy  ts  grantad  providod  this  iwtica  appoars. 
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Appendix  A:  Template  for  SAM  Opening  Briefing,  continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing 


Introduction 


The  template  for  the  final  findings  briefing  provides  ideas  on  how  to 
present  the  findings  resulting  from  the  on-site  period  of  the  appraisal  to 
the  appraisal  participants.  Appraisers  are  welcome  to  design  whatever 
materials  suit  their  style/needs.  The  basic  purpose  of  the  final  findings 
briefing  and  suggested  contents  are  found  in  Section  2.2.14. 


continued  on  next  page 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


continued  on  next  page 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


Scope  of  the  Assessment 

•  <organizational/site  scope> 

•  Assessment  followed  the  SE-CMM  Appraisal 
Method  <or  cite  taiioring> 

•  XX  Systems  Engineering  Leads 

-  <Proj  A> 

-  <Proj  B> 

•  XX  systems  engineers  and  support  personnel 
chosen  from  an  extended  project  set  across 
the  organization 

•  XX  assessment  team  members 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


Conduct  of  the  Assessment 
Process 

Entire  team  highly  responsive 

Strong  consensus  for  systems  engineering 
improvement 

Collaborative  and  enthusiastic  participation 
Candid  data  gathering 


Primary  Assessment  Objectives 

•  During  On-site  Week: 

-  Understand  our  organization’s  current  systems 
engineering  practices 

“  Identify  key  areas  for  process  improvement 
~  Pre-release  training  on  model/appraisal  method 

•  Post  On-site  Week 

-  Develop  findings  and  recommendation  report 

-  Develop  an  action  plan 

-  Management  decision  on  focus  of  process  improvement 
effort 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


Findings  Development  Process, 
continued 

•  Findings  Criteria 

-  We  heard  it 

-  No  sweeping  statements 

“  Only  Issues  with  potential  recommendations 

-  Appraisal  team  consensus 


A-20 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


Process  Area  Ratings 


<insert  profile 
histogram 
or  table  here> 


1=Performed  Informally 
2=Planned  &  Tracked 
3=Defined  Process 
4=Quantitatively  Controlled 
5=Continuously  Improving 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


SE  -  CMM 


Systems  Engineering  improvement 


More 

Model 

Information 
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Appendix  B:  Template  for  SAM  Final  Findings  Briefing,  Continued 


If  you  always  do  what  you've 
always  done 

you'll  always  get  what 
you've  always  gotten! 


continued  on  next  page 
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Appendix  C:  Data  Tracking  Sheet  and  Instructions 


Introduction  The  data  tracking  sheet  (DTS)  is  the  prima^  tool  used  throughout  the 

on-site  period  to  consolidate  and  track  the  information  being  gathered 
from  the  different  data  sources  used  in  the  appraisal.  It  is  a  key  input 
into  the  ratings  and  findings  development  process,  and  when  used 
effectively,  adds  greatly  to  the  team's  ability  to  understand  where 
sufficient  data  have  been  obtained,  and  where  additional  information  is 
still  needed  about  the  practice  of  the  base  and  generic  practices  within 
the  appraised  entity.  The  DTS  is  referred  to  throughout  the  process 
elements  of  the  on-site  period  described  in  Chapter  2. 


Symbols  used  in  Data  in  the  DTS  are  recorded  primarily  via  the  use  of  four  symbols: 

the  DTS 

+  Indicates  that  information  was  heard  or  otherwise  obtained  which 
strengthens  or  corroborates  the  view  that  the  practice  under 
consideration  is  being  performed. 

-  Indicates  that  information  was  heard  or  otherwise  obtained  which 
weakens  or  opposes  the  view  that  the  practice  under  consideration  is 
being  performed. 

?  Indicates  that  information  was  heard  or  otherwise  obtained  in 
relation  to  the  practice,  but  the  nature  of  the  information  was  such 
that  questions  related  to  the  performance  of  the  practice  have  not 
been  answered. 

NA  Indicates  that  the  practice  is  considered  not  to  be  applicable  to  the 
appraised  entity. 

The  use  of  these  symbols  is  explained  in  the  process  element 

descriptions  in  the  main  body  of  the  document,  and  illustrated  in  the 

blocks  below. 


Structure  of  the  The  DTS  is  composed  of  three  pages  per  process  area.  Each  page  lists 
DTS  the  short  titles  of  the  base  practices  for  that  PA  and  the  generic  practices 

for  all  five  capability  levels  as  the  rows  of  a  matrix.  The  columns  of  the 
matrix  provide  space  to  record  the  gathering  of  corroborating  or 
opposing  information  in  relation  to  the  base  and  generic  practices  of  the 
SE-CMM.  Note  that  information  other  than  that  specifically  related  to 
the  model  is  likely  to  be  gathered  in  the  team  notes,  and  may  be  used  to 
create  findings  where  appropriate.  However,  these  data  are  not 
specifically  used  in  the  creation  of  the  rating  and  so  are  not  recorded  in 
the  DTS. 


continued  on  next  page 
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Appendix  C:  Data  Tracking  Sheet  and  Instructions,  continued 


DTS  Contents  Table  C-1  explains  each  of  the  columns  of  the  DTS; 


Column  Title 

Explanation 

Base/Generic 

Practices 

this  column  contains  a  listing  of  the  short  titles 
of  the  base  practices  for  the  listed  PA,  as  well 
as  all  the  generic  practices. 

Questionnaires  A-G 

These  columns  record  the  translation  of  the 
yes/no  responses  from  the  questionnaires  of  the 
respondents  into  -(-/-/?/NA  symbols. 

Interviews  A-D 

It  is  assumed  that  a  maximum  of  four  projects 
will  be  appraised  in  a  single  appraisal.  These 
columns  provide  space  for  recording 
impressions  from  the  initial  interviews  with 
the  SE  leads. 

Practitioner 

Interviews 

These  columns  provide  space  for  recording 
impressions  from  the  practitioner  interviews, 
of  which  there  are  usually  three  focus  groups. 

Preliminary  Rating 

This  the  first  estimate  of  the  rating  of  each 
base  or  generic  practice.  Each  team  member 
uses  the  pattern  of  corroborating  and 
opposing  information  to  formulate  an  initial 
opinion  of  the  performance  of  that  practice  in 
the  appraised  entity  as  a  whole. 

Preliminary  Findings 
Review  A-D 

These  columns  provide  space  for  recording 
impressions  from  the  preliminary  findings 
review  with  the  SE  leads. 

Draft  Rating 

This  column  is  where  the  preliminaiy  rating  is 
refined  with  data  gathered  via  the  prelinoinaiy 
findings  review,  and  is  the  rating  that  will  be 
correlated  with  the  findings  development. 

Findings  Reviews 

These  columns  provide  space  for  recording 
information  gathered  during  the  draft  findings 
reviews  by  the  SE  leads  and  practitioners. 

Final  Rating 

This  is  the  final  rating  for  each  base/generic 
practice  upon  which  team  has  reached 
consensus,  and  is  the  basis  for  the  profile  that 
is  presented  in  the  final  findings  briefing. 

Table  C-1.  DTS  Contents. 


continued  on  next  page 
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Appendix  C:  Data  Tracking  Sheet  and  Instructions,  continued 


Diagram  of  an 
example  DTS 


A  DTS  can  be  created  in  a  spreadsheet,  database,  or  other  tool 
compatible  with  the  appraiser’s  environment.  The  following  diagram 
provides  an  example  of  a  DTS  created  in  a  spreadsheet  environment. 


Questionnaires 

Initial 

A 

B 

C 

D 

E 

F 

G 

Projected 

Rating 

PA  01 :  Analyze  Candidate  Solutions 

Base  Practices: 

1.1 

Establish  Evaluation  Criteria 

1.2 

Define  Approach 

1.3 

Identify  Additional  Alternatives 

1.4 

Analyze  Candidate  Solutions 

1.5 

Select  Solutions 

1.6 

Capture  Results 

Generic  Practices: 

2.1.1 

Allocate  Resources 

2.1.2 

Assign  Responsibilities 

2.1.3 

Document  Approach 

2.1.4 

Provide  Tools 

2.1.5 

Ensure  Training 

2.1.6 

Plan  Performance 

2.2.1 

Follow  Plans 

2.2.2 

Do  Configuration  Management 

2.3.1 

Verify  Process  Compliance 

2.3.2 

Audit  Work  Products 

2.4.1 

Track  with  Measurement 

2.4.2 

Take  Corrective  Action 

3.1.1 

Standardize  Process 

3.1.2 

Tailor  Standard  Process 

3.2.1 

Follow  Defined  Process 

3.2.2 

Perform  Peer  Reviews 

3.2.3 

Use  Process  Data 

4.1.1 

Establish  Quality  Goals 

4.2.1 

Determine  Process  Capability 

4.2.2 

Use  Process  Capability 

5.1.1 

Establish  Process  Goals 

j 

1 

5.1.2 

Improve  Defined  Process 

5.2.1 

Do  Causal  Analysis 

5.2.2 

Eliminate  Defect  Causes 

5.2.3 

Improve  Standard  Process 

Figure  C-1.  DTS  Example  Diagram. 


continued  on  next  page 
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Appendix  C:  Data  Tracking  Sheet  and  Instructions,  Continued 


Diagram  of  an 
example  DTS, 
continued 


Interviews 

Prelim 

A 

B 

C 

D  . 

Gpl 

Gp2 

Gp3 

Rating 

PA 

01:  Analyze  Candidate  Solutions 

n 

Base  Practices: 

l.l 

Establish  Evaluation  Criteria 

1. 2 

Define  Approach 

1. 3 

Identify  Additional  Alternatives 

1.4 

Analyze  Candidate  Solutions 

1.5 

Select  Solutions 

1 .6 

Capture  Results 

Z[ 

Generic  Practices: 

<sanie  as  above> 

Findings  Review 

Final 

A 

B 

C 

D 

0th 

0th 

0th 

Rating 

PA 

01:  Analyze  Candidate  Solutions 

n 

Base  Practices: 

l.l 

Establish  Evaluation  Criteria 

1.2 

Define  Approach 

1. 3 

Identify  Additional  Alternatives 

1.4 

Analyze  Candidate  Solutions 

1.5 

Select  Solutions 

1.6 

Capture  Results 

z 

Generic  Practices: 

<same  as  above> 

Figure  C-1.  DTS  Example  Diagram,  continued 
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Appendix  D:  Sample  Schedule  for  the  On-Site  Week 


Introduction 


SAM  Sample 
Schedule 


A  sample  on-site  schedule  is  provided  in  this  section.  This  schedule  is 
based  on  a  sample  of  three  projects  for  the  appraisal  and  three  sets  of 
practitioner  groups. 


SAM  -  Sample  Schedule  | 

Start 

Finish 

Dur. 

Description 

Day#! 

8:00  AM 

9:30  AM 

1:30 

Ooeninq  meetinq 

9:30  AM 

1 2:00  PM 

2:30 

Brief  team 

12:00  PM 

1 :00  PM 

1:00 

Lunch 

1:00  PM 

5:00  PM 

4:00 

Analyze  questionnaire  and  develop  response 

Day  #2 

8:00  AM 

9:30  AM 

1:30 

Interview  SE  lead  #1 

9:30  AM 

1 1 :00  AM 

1:30 

Interview  SE  lead  #2 

1 1 :00  AM 

12:30  PM 

1:30 

Interview  SE  lead  #3 

12:30  PM 

1 :30  PM 

1:00 

Lunch 

1:30  PM 

3:30  PM 

2:00 

Interview  practioner  group  #1 

3:30  PM 

5:00  PM 

1:30 

Consolidate  data 

Day  #3 

8:00  AM 

10:00  AM 

2:00 

Interview  practioner  qroup  #2 

10:00  AM 

1 2:00  PM 

2:00 

interview  practioner  group  #3 

12:00  PM 

1 :00  PM 

1:00 

Lunch 

1 :00  PM 

2:30  PM 

1:30 

Consolidate  data 

2:30  PM 

Develop  oreliminarv  findinqs 

Day  #4 

8:00  AM 

9:00  AM 

1:00 

Review  preliminary  findinqs  with  SE  lead  #1 

9:00  AM 

10:00  AM 

1:00 

Review  preliminary  findinqs  with  SE  lead  #2 

10:00  AM 

1 1 :00  AM 

1:00 

Review  preliminary  findings  with  SE  lead  #3 

1 1 :00  AM 

1  2:00  PM 

1:00 

Consolidate  data 

1 2:00  PM 

1 :00  PM 

1:00 

Lunch 

1 :00  PM 

Draft  findinqs  and  ratinqs 

Day  #5 

8:00  AM 

9:00  AM 

1:00 

Present  draft  findinqs  to  practioners 

9:00  AM 

10:00  AM 

1:00 

I  Present  draft  findinqs  to  SE  leads 

10:00  AM 

1  2:00  PM 

2:00 

!  Adjust  draft  findings 

1 2:00  PM 

1 :00  PM 

1:00 

;  Lunch 

1 :00  PM 

2:30  PM 

1:30 

Present  final  briefinq 

2:30  PM 

3:30  PM 

1:00 

Conduct  executive  session 

3:30  PM 

4:30  PM 

1:00 

Wrap-up 

Figure  0-1.  Sample  SAM  Schedule. 
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Appendix  E:  SAM  Training  Support 


Introduction 


The  SE-CMM  Project  has  not  committed  to  producing  training  materials 
for  SAM.  However,  the  project  recognizes  that  some  training  beyond 
similar  organizational  appraisal  methods  is  necessary  to  prepare 
appraisal  team  members.  Many  of  the  materials  provided  in  the  SAM 
appendices  can  be  viewed  as  training  support  materials.  The 
information  sheets  provided  in  this  appendix  summarize  the  steps  of  the 
on-site  period  for  SAM  and  provide  an  easy  reference  for  appraisers  to 
determine  the  sequence  of  events. 
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Goals 


Conduct  Opening  Meeting 


—  Visibly  demonstrate  senior  management  support  for  the  assessment  process. 

-  Introduce  principles  of  process  management. 


—  Review  schedule  for  the  assessment  period. 

Participaiits:  Ssiior  ManageDient  +  A^essmentteam  4-  SE  leads  4-Rractitioners 

Duration:  LSfeoar 


Approach 


The  senior  site  manager  begins  this  meeting  by  welcoming  the  assessment  team  and  indicating 
management  support  for  the  assessment  process.  The  assessment  team  leader  delivers  a  brief 
presentation  on  process  management  and  the  assessment  process  flow.  Finally,  the  site 
coordinator  reviews  the  assessment  schedule,  reminding  everyone  of  when  and  where  they  are 
expected  to  participate. 

□  Senior  management  opening. 

-  Introduce  and  welcome  assessment  team. 

-  Indicate  support  for  assessment  and  process  improvement. 

-  Solicit  full  support  and  participation. 

□  Briefing  on  process  management  principles. 

□  Briefing  on  assessment  process  flow. 

-  Stress  openness. 

-  Emphasize  confidentiality. 

□  Review  of  assessment  schedule. 

Q  Question  and  answer  period. 
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Familiarize  Team  with  SAM 


Goals 

-  Introduce  the  team  to  the  assessment  process. 

-  Prepare  for  the  discussions  with  the  SE  leaders  and  practitioners. 

-  Review  the  answers  to  the  questionnaire. 

:  ~  yK?  V’-  ^ JVj,,  '  i 

This  is  an  opportunity  for  the  assessment  team  to  begin  to  work  together.  The  assessment 
steps  are  presented  in  greater  detail  than  at  the  opening  meeting,  and  the  team's  role  in  each  step 
is  clarified.  The  assessment  work  begins  with  a  review  of  the  answers  to  the  questionnaire 
along  with  the  exploratory  questions  for  the  project  leaders.  The  exploratory  questions  may  be 
tailored  where  special  conditions  are  identified. 

Approach 

□  Team  building  exercise  (optional). 

Q  Review  of  assessment  steps. 

-  Explain  conduct  of  SE  leader  discussions. 

-  Explain  conduct  of  practitioner  discussions. 

□  Examine  exploratory  questions  and  answers  to  the  questionnaire. 

□  Review  assessment  schedule. 

Q  Question  and  answer  period. 
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Goals 


Analyze  Questionnaire 


-  Review  the  answers  to  the  questionnaire. 

-  Develop  exploratory  questions. 

-  Prepare  for  discussions  with  the  SE  leader. 

Partklpantsi  Assessment  team 

Duration: hours'.- '  'S  ■ 

The  assessment  begins  with  a  review  of  the  answers  to  the  questionnaire.  Based  on  the 
answers  to  the  questionnaire,  the  team  develops  exploratory  questions.  Approximately  40-50 
questions  should  be  generated  for  each  project  lead.  The  questions  should  be  designed  to  elicit 
more  then  just  a  "yes/no"  response.  For  example,  questions  often  begin  with,  “Would  you 
please  describe  . . .”  When  developing  a  question  we  are  often  looking  for  certain  responses, 
e.g.,  SE  management  plan  .  Note  these  words  as  “listen  fors,"  which  the  facilitator  can  use 
as  a  cue  to  ask  additional  questions  if  they  are  not  mentioned.  Some  questions  may  be 
accompanied  by  a  request  for  relevant  or  supporting  documents. 

Questions  should  be  used  to  refine  answers  or  explore  inconsistencies.  There  is  a  separate  set 
of  questions  for  each  SE  lead.  However,  there  is  usually  some  overlap.  Once  the  questions 
are  developed,  they  should  be  transcribed  onto  the  appropriate  form,  and  copies  made  for  each 
team  member.  The  form  should  include  each  question,  any  “listen  fors,"  document  requests, 
and  room  for  notes. 

Approach 

Review  answers  to  the  questionnaire. 

Brainstorm  a  preliminary  set  of  questions. 

Organize  and  eliminate  redundant  questions,  add  “listen  fors". 

Transcribe  questions  onto  forms  and  make  copies  for  team. 

Review  next  day’s  schedule. 


□ 

□ 

□ 

□ 

□ 
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Interview  SE  Leads 


Goals 

—  Resolve  any  misunderstandings  from  the  questionnaire. 

—  Clarify  any  anomalies  or  inconsistencies. 

—  Focus  assessment  team  on  process  areas  that  need  improvement. 

Approach 

The  team  leader  will  conduct  the  discussions.  All  other  team  members  should  take  notes.  Any 
team  member  can  ask  a  question,  although  the  team  leader  should  be  allowed  to  lead  the 
questioning  to  ensure  that  all  high-priority  questions  are  covered.  The  team  leader  should 
follow-up  on  the  “listen  fors"  and  note  any  document  requests. 

□  Introduce  SE  lead  and  the  team. 

O  Repeat  confidentiality  rules 

—  No  individuals  or  projects  named  in  results. 

—  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

—  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

Q  Explain  the  purpose  of  this  session. 

D  Cover  the  material  in  the  exploratory  questions. 

□  Remind  SE  lead  of  any  document  requests. 

Q  Remind  SE  lead  of  the  time  and  place  for  the  preliminary  findings  review. 

□  Thank  and  excuse  SE  lead. 
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Consolidate  Data  from  SE  Lead  Interviews 


Goals 


—  Give  team  members  a  chance  to  reflect  on  previous  sessions. 

—  Discuss  any  confusing  or  missing  information. 

-  Perform  adjustments  to  up-coming  activities. 

Participants:  ;  Assessment  team 
DuraiitMi:  1.0  to  1.5  hours 

Approach 

These  sessions  give  the  team  a  chance  to  absorb  the  data  they  have  been  given  in  previous 
sessions.  They  begin  by  reviewing  their  notes  and  then  discuss  any  confusing  or  missing  data. 
Next,  changes  (if  any)  to  upcoming  activities  are  discussed. 

Q  Quietly  review  notes. 

Q  Discuss  issues. 

Q  Adjust  upcoming  activities. 
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Interview  Practitioners 


Goals 

—  Hear  concerns  and  views  from  the  perspective  of  the  practitioners. 

-  Verify  information  gathered  from  questionnaire  and  discussions  with  the  SE  leaders. 
—  Listen  for  new  process  issues  or  areas  of  concern. 

Approach 

Each  group  consists  of  6-10  professionals  considered  to  be  experts  and  opinion  leaders  in  the 
organization.  They  should  be  actually  working  on  projects  (i.e.,  not  staff  or  management). 
The  team  leader  opens  and  closes  each  session  and  facilitates  the  discussion.  Assessment  team 
members  need  to  relax  and  let  the  discussion  flow:  do  not  lead  the  discussions.  All 
assessment  team  members  should,  however,  take  notes  during  these  discussions.  Team 
members  should  occupy  every  other  seat  around  the  table  in  order  to  avoid  an  “us  vs.  them" 
atmosphere. 

□  Introduce  the  topic  (if  appropriate). 

Q  Explain  the  conduct  of  the  meeting. 

-  How  the  operation  looks  to  them. 

-  Free-form  discussion  (team  will  not  ask  specific  questions). 

-  Chance  to  summarize  your  major  issues  at  the  end. 

Q  Repeat  confidentiality  rules. 

-  No  individuals  or  projects  named  in  results. 

-  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

—  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

□  Introduce  everyone  (state  name  and  function). 

G  Turn  meeting  over  to  the  practitioners. 

□  About  20-30  minutes  from  the  end  (5  minutes  per  person),  ask  each  practitioner 

If  you  could  change  one  thing  in  your  organization  other  then  your  boss  or  your 
paycheck,  what  would  it  be? 

Other  than  the  people,  what  do  you  think  is  this  organization's  major  strength? 

G  Remind  practitioners  of  the  time  and  place  for  the  draft  briefing  presentation. 

G  Thank  and  excuse  the  practitioners. 
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Develop  Preliminary  Findings 


Goals 


—  Identify  list  of  key  process  issues. 

-  Generate  preliminary  findings. 

Participaiits;  'Assmmeiitfe^oiily'.  . 

;Dura|ipn:_  ;3-9  hours  • 

Approach 

Focus  on  issues  for  the  entire  organization.  It  is  important  that  the  findings  have  the  broadest 
possible  application,  both  to  preserve  confidentiality  and  have  maximum  impact  with  senior 

management.  Try  to  keep  the  goals  in  mind  and  use  the  team's  expertise  to  solve  the  problems 
Avoid  the  following: 

-  Issues  without  useful  recommendations. 

—  Findings  based  on  hearsay. 

-  Sweeping  statements. 

Formulating  the  findings  is  the  most  difficult  part  of  the  assessment.  We  begin  with 
preliminary  findings  which  are  40-60  simple  statements.  At  this  point  in  the  assessment,  it  is 
not  necessary  to  have  team  consensus.  Nor  is  it  necessary  to  word  smith  the  findings  although 
redundant  questions  should  be  consolidated. 

Brainstorm  a  list  of  findings  -  organized  by  PA. 

Review  each  PA  with  issues  and  eliminate  redundant  questions. 

Transcribe  list  onto  form  and  make  copies  for  each  team  member. 


□ 

□ 

□ 
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Review  Preliminary  Findings 


Goals 

-  Get  feedback  on  preliminary  findings. 

-  Collect  any  requested  documentation. 

-  Ask  SE  leads  for  strengths  and  weakness. 

.Parfkigaiitstf^ 

Approach 

The  team  leader  will  conduct  the  session,  and  all  other  team  members  should  take  notes.  Any 
team  member  can  ask  a  question,  although  the  team  leader  should  be  allowed  to  lead  the 
questioning  to  ensure  that  ^1  preliminary  findings  are  covered. 

□  Re-introduce  SE  lead  and  the  team. 

Q  Repeat  confidentiality  rules. 

-  No  individuals  or  projects  named  in  results. 

-  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

-  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

□  Collect  any  requested  documents. 

Q  Explain  the  purpose  of  this  session. 

□  '  State  each  preliminary  finding  and  ask: 

Is  the  finding  true  for  your  project? 

Is  the  finding  true  for  the  organization? 

□  About  five  minutes  from  the  end  ask: 

If  you  could  change  one  thing  in  your  organization  other  then  your  boss  or  your 
paycheck,  what  would  it  be? 

Other  then  the  people,  what  do  you  think  is  this  organization's  major  strength  ? 

Q  Remind  SE  lead  of  the  time  and  place  for  the  draft  findings  presentation, 
d  Thank  and  excuse  SE  lead. 
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Goals 


Develop  Draft  Rating 


Establish  team  consensus  on  process  capability  profile. 

-  Produce  process  capability  profile  for  presentation  in  Final  Briefing. 

As^ssment  team  V 

Approach 


As  a  result  of  the  data  gathering  taken  place  so  far,  team  members  should  have  updated  DTSs 
that  reflect  the  data  provided  via  the  questionnaires,  interviews,  and  any  documentation  reviews 
or  presentations  conducted.  The  profile  is  determined  prior  to  synthesizing  the  draft  finds  so  as 
to  mtorin  the  team's  decision  making  on  how  to  prioritize  the  findings.  The  findings  should 
provide  the  7-9  top  issues  that  improvement  should  focus  on-without  the  rating  process  being 
relatively  complete,  issues  which  may  have  voluble  supporters  but  do  not  provide  significant 
leverage  points  could  creep  into  the  findings.  The  rating  algorithm  for  SAM  is  relatively 
simple:  100%  of  applicable  base  practices  should  be  exhibited  throughout  the  entire  sample  of 
projects  selected  for  a  rating  of  "1"  to  be  achieved,  and  80-90%  (depending  on  the  capability 
level)  of  the  generic  practices.  t'  j 

Q  Step  through  each  process  area. 

-  Review  notes  from  interviews 


-  Apply  rating  algorithm. 

-  Obtain  team  consensus  on  process  area  rating. 

□  Review  profile  as  a  whole  for  consistency. 

□  Determine  presentation  style  for  profile  and  prepare  for  the  final  briefing. 


Develop  Draft  Findings 


Goals 


-  Assign  a  level  of  practice  for  each  PA. 

-  Identify  list  of  (7  ±  2)  key  process  issues 

-  Generate  draft  findings  briefing. 


Approach 

The  assessment  team  will  assign  a  level  of  practice  (0-5)  for  each  PA.  Use  the  questionnaire 
analysis  and  notes  from  the  discussions  to  assist  in  this  process. 

Focus  on  issues  for  the  entire  organization.  It  is  important  that  the  findings  have  the  broadest 
possible  application,  both  to  preserve  confidentiality  and  have  maximum  impact  with  senior 
management.  Limit  the  number  of  findings  to  7  ±  2.  This  gives  definite  direction  for  process 
improvement  without  overloading  limited  resources.  Each  finding  should  consist  of 

Finding  -  A  single  statement  of  the  issue. 

Causes  -  Observations  that  contribute  to  the  finding. 

Consequences  -  Results  that  will  get  management  attention,  e.g.,  increased  rework. 

Preliminary  findings  are  often  found  to  be  causes  of  a  more  general  finding.  Both  findings  and 
causes  should  reflect  what  the  team  has  heard  from  the  SE  leads  and  the  practitioners.  The 
consequences  are  developed  by  the  team  to  ensure  management  attention,  and  need  not 
represent  consequences  voiced  by  the  SE  leader  or  the  practitioners.  Again,  avoid  issues 
without  useful  recommendations,  unsubstantiated  findings,  or  sweeping  generalizations. 

Formulating  findings  is  the  most  difficult  part  of  the  assessment.  It  is  the  place  where  the  team 
is  most  likely  to  have  conflict.  Try  to  keep  the  SE-CMM  in  mind  and  use  the  team’s  expertise 
to  solve  the  problems.  For  each  finding,  the  team  should 

Q  Identify  level  of  practice  for  each  PA. 

□  Review  notes  and  discussions  of  preliminary  findings. 

□  Refine  issues  into  findings,  causes,  and  consequences. 

□  Form  mini-teams  to  word  smith  individual  findings. 

□  Review  each  finding  to  reach  consensus  on  wording  (entire  team). 

Q  Complete  briefing  and  produce  overheads  for  presentation. 
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Present  Draft  Findings 


Goals 

—  Provide  SE  leads  and  practitioners  with  the  opportunity  to  comment  on 
findings. 

—  Allow  the  team  to  judge  the  impact  of  the  findings  on  the  organization. 

-  Build  organizational  momentum  for  process  improvement. 

Parficipaiits:  Assessment  team  -f-  Practitioners.  Assessment  team  -i-  SE  leads 

{separate  sessions) 

Daration: _  1  hour  for  each  of:  ^ 

Approach 

The  assessment  team  leader  presents  the  findings  to  the  practitioners  and  SE  leads  in  separate 
sessions,  first  to  the  practitioners,  and  then  to  the  SE  leads.  These  groups  are  kept  separate  so 
opening  comments  are  not  inhibited.  The  ratings  are  not  presented  at  this  time.  In  each  session, 
the  findings  are  first  presented  without  interruption  so  that  the  audience  has  a  chance  to  hear  all 
of  the  findings.  The  presenter  should  use  the  exact  wording  from  the  slides.  The  presenter 
then  steps  through  each  finding  and  asks  for  comments.  The  assessment  team  members  may 
assist  the  leader  in  explaining  any  issues,  but  should  concentrate  on  taking  notes. 

Ql  Welcome  participants  and  set  the  context. 

CH  Repeat  confidentiality  rules. 

—  No  individuals  or  projects  named  in  results. 

—  They  may  not  disclose  comments  of  others  made  at  this  meeting. 

—  Team  will  take  notes.  All  notes  will  be  treated  as  confidential. 

Q  Make  presentation  (without  interruption). 

Q  Repeat  each  finding  and  solicit  comments. 

□  Remind  participants  of  the  time  and  place  of  the  final  presentation. 
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Adjust  Draft  Findings 


Goals 


-  Refine  wording  for  final  findings  presentation. 


-  Prepare  final  presentation. 


2  hours 


Approach 

As  a  result  of  hearing  the  findings  presentation  and  hearing  the  two  groups'  comments,  the 
team  will  see  places  where  the  thrust  or  wording  of  some  of  the  findings  needs  refinement. 
The  goal  should  be  to  maximize  the  acceptance  of  the  assessment.  It  is  important  to  use  the 
assessment  to  build  momentum  for  process  improvement.  This  sometimes  requires  weakening 
or  strengthening  the  wording  for  a  finding. 

Therefore,  a  set  of  next  steps  should  be  scheduled  that  build  on  the  findings  and  result  in 
observable  changes.  If  the  organization's  expectations  are  not  satisfied,  the  opportunity  for 
change  may  be  lost  forever! 

d  Step  through  each  finding. 

-  Review  notes  from  presentations. 

-  Refine  wording. 
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Goals 


Present  Final  Briefing 


—  Visibly  present  the  results  of  the  assessment  to  senior  management. 

—  Build  support  for  addressing  the  findings. 

-  Review  next  steps. 

Participants:  Sehibf  niahagement  +  Asfessment  team  ^  SE  leads + PractMoneis 

'I>nratinn;_ . _“;2homs_  _  ,  __ 

Approach 

The  assessment  team  leader  presents  the  final  findings  brief  The  final  briefing  will  include 

1.  Assessment  scope  -  projects  &  participants  (thank  everyone!). 

2.  Rating  -level  of  maturity  for  each  PA. 

3.  Strengths  -  organization’s  strong  points. 

4.  Findings  -  adjusted  draft  findings. 

5.  Next  steps  -  findings  &  recommendations  report,  action  plan, . .  . 
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Brief  Sponsor  (optional) 


Goals 


-  Provide  additional  background. 

-  Resolve  any  open  issues  with  senior  management. 

-  Discuss  next  steps. 

wWlo ' ' 

Approach 

This  optional  session  gives  senior  management  an  opportunity  to  ask  questions  and  discuss 
any  issues  that  they  were  reluctant  to  raise  in  the  open  form  of  the  final  briefing.  It  is  also  an 
opportunity  for  the  team  leader  (or  the  entire  team)  to  promote  follow-on  activities.  Remember 
that  the  confidentiality  rules  still  apply!  Do  not  let  senior  management  use  this  session  to  fix 
blame  for  any  problem  or  to  attribute  particular  findings  to  a  project  or  individual. 
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Conduct  Wrap-up 


Goals 

-  Evaluate  the  SE-CMM. 

—  Evaluate  the  assessment  process. 

-  Plan  next  steps. 

Participants:  Assessment  team  ' 

Dnrafipa;_  _-lhpur_,.. . ■  . 

Approach 

The  assessment  team  uses  this  session  to  generate  feedback  on  the  SE-CMM  pilot  assessment. 
Each  team  member  completes  the  questionnaire  evaluation  form.  These  results,  along  with  the 
questionnaire  evaluation  form  from  the  SE  leads  is  returned  to  the  SE-CMM  authors  for 
review.  Next,  a  discussion  focuses  on  the  model  and  the  assessment  process. 

Each  team  member  completes  a  questionnaire  evaluation  form. 

The  team  discusses  the  SE-CMM.  What  works?  What  doesn't?  What's  missing? 

The  team  discusses  what  in  the  assessment  process  worked  and  what  did  not. 

Before  the  team  breaks,  the  next  steps  should  be  scheduled  and  responsibilities  assigned. 

d  Discuss  next  steps. 

-  Findings  and  recommendations  report? 

—  Action  plan? 

□  Schedule  next  steps. 


□ 

□ 

□ 
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Appendix  F:  Site  Coordination  Checklist 


Introduction 


Preparation  tasks 
(prior  to  on-site 
period) 


This  checklist  is  used  to  support  the  SE-CMM  appraisal  site  coordinator 
in  preparing  for  the  on-site  period  of  the  SE-CMM  appraisal  method 
(SAM). 


Major  events  that  the  site  coordinator  is  responsible  for  arranging  are 
described  in  the  table  below.  The  time  frames  given  are  approximate, 
and  are  based  around  the  beginning  of  the  on-site  period.  The 
coordinator  can  use  this  list  as  a  checklist  for  preparation. 


V 

Task 

Description 

Time  frame 

Executive 
briefing  to  obtain 
sponsorship  for 
SAM 

Briefing  by  site  coordinator  or 
facilitator,  as  appropriate,  to  the 
potential  sponsor  introducing 
the  SE-CMM  and  SAM 
concepts. 

At  least  six 
weeks  prior 
to  on-site 
period 

Determine 

confidentiality 

requirements 

Determine  with  senior 
management  the  need  for 
nondisclosure  and 
confidentiality  of  agreements. 

Four  to  six 
weeks  prior 
to  on-site 
period 

Select  projects 
(three  to  four 
projects) 

Site  coordinator,  working  with 
the  sponsoring  manager,  selects 
projects  appropriate  to  the 
appraisal  purpose. 

Four  to  six 
weeks  prior 
to  on-site 
period 

Determine  on¬ 
site  week 

Working  with  senior 
management  and  the  facilitator, 
determine  a  week  when  the 
systems  engineering  leads, 
senior  management,  facilitators, 
and  potential  appraisal  team 
members  are  available  as 
needed. 

Four  to  six 
weeks  prior 
to  on-site 
period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator. 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation  tasks, 
continued 


V 

Task 

Description 

Time  frame 

Select 
appraisal 
team  (five  to 
seven  team 
members 
plus  one  to 
two 

facilitators) 

Site  coordinator  typically 
determines  the  pool  of 
appraisal  team  members  for 
consideration  by 
management,  unless  the 
coordinator  has  been 
empowered  to  make  the 
selection,  in  which  case  the 
selections  are  made.  The 
facilitator  is  usually  available 
to  help  in  screening 
candidates.  Provide  a  copy 
of  the  SE-CMM  and  SAM 
description  to  the  appraisal 
team  members. 

Four  to  six 
weeks  prior  to 
on-site  period; 
goal  is  to  select 
the  team  far 
enough  in 
advance  to  be 
able  to  get  the 
team  members 
to  schedule  the 
appraisal  week 

Select 

systems 

engineering 

lead 

For  each  project  selected, 
identify  the  systems 
engineering  leads  for  the 
selected  projects  and  talk  to 
them  about  their 
involvement.  Also,  verify 
their  availability  during  the 
appraisal  week 
(approximately  six  hours  per 
person). 

Three  to  four 
weeks  prior  to 
on-site  period 

Select 

practitioners 

Select  potential  interviewees 
in  the  selected  projects  as 
well  as  other  projects  in  the 
organization  being  appraised. 
Typically,  the  facilitator  is 
available  to  consult  on 
participants,  and  management 
typically  approves  the 
participant  list. 

Three  to  four 
weeks  prior  to 
on-site  period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 


continued  on  next  page 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation  tasks, 
continued 


V 

Task 

Description 

Time  frame 

Administer 

questionnaire 

The  selected  systems 
engineering  leads  (and  other 
practitioners,  if  selected  for 
questionnaire  analysis)  are 
provided  with  the  SAM 
questionnaire,  and  the  site 
coordinator  provides  the  time 
frame  for  returning  the 
questionnaire.  Hand  out  the 
SE-CMM  glossary  with  the 
questionnaire.  The  site 
coordinator  should  be 
available  to  answer  any 
clarifying  questions;  usually 
a  one  or  two  day  turnaround 
is  requested  if  the 
questionnaires  are  not 
completed. 

Two  weeks 
prior  to  on-site 
period 

Handout 

about 

questionnaire 

For  pilots,  when  the 
questionnaire  is  handed  out, 
also  hand  out  the 
questionnaire  about  the 
questionnaire  and  emphasize 
the  need  for  feedback  from 
people  who  answer  the 
questionnaire. 

Two  weeks 
prior  to  on-site 
period 

Collect 

questionnaire 

The  site  coordinator  collects 
the  completed  questionnaires, 
makes  a  copy  for  disaster 
recovery  purposes,  and  mails 
questionnedres  back  to  the 
facilitator  for  initial  analysis. 

To  be  received 
by  facilitator 
one  week  prior 
to  on-site 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation  tasks, 
continued 


V 

Task 

Description 

Time  frame 

Prepare 

appraisal 

team 

notebook 

Prepare  a  three-hole  binder 
for  each  appraisal  team 
member  wMch  contains 
copies  of  the  completed 
questionnaires  and  blank 
paper  for  taking  notes.  Tabs 
for  each  project,  each 
practitioner  discussion, 
preliminary  findings,  and 
findings  briefing  are  often 
helpful.  These  notebooks  are 
handed  out  during  the  team 
training  at  the  beginning  of 
the  on-site  period. 

One  week  prior 
to  on-site  period 

Schedule 
rooms  for 
opening  and 
final  briefing 
rooms 

Schedule  rooms  large  enough 
to  hold  all  anticipated 
participants,  including 
management,  appraisal 
participants,  and  other 
identified  invitees. 

Two  weeks 
prior  to  on-site 
period, 
depending  on 
how  tight 
facilities  are 

Schedule 
rooms  for 
practitioner 
interviews 

Schedule  rooms  large  enough 
to  hold  20  people  for  the 
practitioner  interviews. 

Two  weeks 
prior  to  on-site 
period, 
depending  on 
how  tight 
facilities  are 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation  tasks, 
continued 


V 

Task 

Description 

Time  frame 

Schedule 
rooms  for 
systems 
engineering 
lead 

interviews 

and 

assessment 

team 

Schedule  a  room  large 
enough  for  10  people  plus 
some  extra  workspace  which 
can  be  blocked  off  for  team 
use  for  the  entire  week  of  the 
on-site  period.  Preferably, 
this  area  is  somewhere  that 
can  be  locked  at  night,  but 
not  in  an  area  that  will  require 
escort  of  the  appraisal  team 
members/facilitators  during 
the  week. 

Two  to  three 
weeks  prior  to 
on-site  period, 
depending  on 
how  tight 
facilities  are 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation  tasks, 
continued 


Task 

Description 

Time  frame 

Schedule 

support 

facilities 

The  following  support  tools 

should  be  provided; 

•  If  possible,  a  personal 
computer  of  the  type  most 
often  used  in  the 
organization  (usually 
Windows-based  or  Mac) 
and  an  associated  dedicated 
laser  printer  scheduled  for 
the  week.  (Desired 
software  is  MS  Word  and 
MS  PowerPoint  for  current 
facilitator  group.) 

•  Instmctions  for  obtaining 
photocopies  in  the  building 
being  used.  (This  is  not 
much  of  an  issue  if  a 
building  familiar  to  the 
appraisal  team  members  is 
being  used.) 

•  Lists  of  facilities,  hotels, 
and  restaurants  that  are 
close  by  (preferably  with  a 
map)  and  information  on 
restaurant  delivery  service. 

•  Flip  charts/markers, 
transparencies,  notepads, 
power  strips,  3-hole 
punch,  binders,  lots  of 
post-it  notes. 

Computer  may 
take  several 
weeks  to 
arrange;  other 
arrangements 
just  need  to  be 
made  prior  to 
on-site  period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 
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Appendix  F:  Site  Coordinator  Checklist,  continued 


Preparation  tasks, 
continued 


V 

Task 

Description 

Time  frame 

Schedule 
support  staff 

If  possible,  provide  dedicated 
secretarial  support  for  the  last 
three  days  of  the  on-site 
period  to  provide 
transcription,  revision, 
printing,  reproduction  and 
note-taking  services. 

Two  to  three 
weeks  prior  to 
on-site  period 

(In  many  cases  the  appraisal 
team  ends  up  doing  their  own 
support,  but  having  a 
secretary  to  do  these  tasks 
can  really  relieve  some  of  the 
grunge  work.) 

Verify  senior 
management 
schedule 

Verify  that  senior 
management  is  prepared  to 
attend  and  speak  at  the 
opening  briefing,  and  attend 
the  fin^  briefing. 

Two  weeks, 
then  one  week 
prior  to  on-site 
period 

Verify 

participant 

schedule 

Verify  that  participants  are 
available  in  their  dotted  time 
slots;  make  any  revisions 
necessary  to  the  schedule. 

Two  weeks, 
then  one  week 
prior  to  on-site 
period 

Verify  team 

member 

schedule 

Verify  that  appraisal  team 
members  have  no  conflicts 
during  the  on-site  period. 

Two  weeks, 
then  one  week 
prior  to  on-site 
period 

Table  F-1.  Preparation  Tasks  for  Site  Coordinator,  continued 
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Appendix  G:  Approved  SAM  Requirements 


Introduction 


This  appendix  contains  the  requirements  for  SAM  approved  by  the  SE- 
CMM  Steering  Group. 


continued  on  next  page 
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Appendix  G:  Approved  SAM  Requirements,  continued 


Introduction 


1.0  Scope 


2.0  Applicability 


3.0  Pre-on-site 
work 


The  following  requirements  are  synthesized  from  the  "Sources  of 
Requirements"  v.3  used  at  the  3/8/94  SE-CMM  author's  meeting.  In 
addition,  some  requirements  (i.e.,  those  in  6.0  and  7.0)  are  derived 
from  the  SEI's  Common  Appraisal  Framework  (CAE),  with  which  the 
method  is  intended  to  be  compatible.  Ultimate  compatibility  with  the 
CAE  depends  on  the  final  form  of  that  framework. 


1.1  The  1994  SE-CMM  Appraisal  Method  (SAM)  scope  is  limited  to 
assessment  of  the  appraised  entity's  process  capability. 

1 .2  The  SAM  deals  with  the  diagnostic  phase  of  a  process  improvement 
program. 

1.3  vl  .0  of  the  SAM  is  focused  on  appraisal  to  support  self- 
improvement. 


The  SAM  will  be 

2.1  Applicable  to  multiple  types  of  appraised  entities  (e.g.,  sites, 
organizations,  and  projects). 

2.2  Adaptable  to  in-house  process  improvement  situations. 

2.3  Adaptable  to  supplier  selection  situations. 

2.4  Applicable  to  contract-driven  environments. 

2.5  Applicable  to  market-driven  environments 


3. 1  Pre-on-site  work  for  participants  of  SAM  will  be  limited  to  four 
hours  per  appraisal  participant. 

3.2  Pre-on-site  work  for  SAM  appraisal  team  members  will  be  limited 
to  40  hours  per  appraisal  team  member,  not  including  training. 

3.3  SAM  appraisal  team  members  will  be  trained  prior  to  participation 
in  an  SE-CMM  appraisal. 


continued  on  next  page 
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Appendix  G:  Approved  SAM  Requirements,  continued 


4.0  On-site  work 


5.0  Post  on-site 
work 


4. 1  On-site  work  for  appraisal  participants  will  be  limited  to  five 
calendar  days,  of  which  not  more  than  two  staff-days  per  person 
will  be  required  over  that  time. 

4.2  SAM  will  target  8-10  hour  days  for  skilled  appraisal  team 
members. 


5. 1  Post-on-site  work  for  appraisal  participants  will  be  limited  to  four 
hours  per  participant. 

5.2  Post-on-site  work  for  appraisal  team  members  to  complete  the 
SAM-related  work  will  be  limited  to  40  hours  per  team  member. 


continued  on  next  page 
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Appendix  G:  Approved  SAM  Requirements,  Continued 


6.0  CAF 
conformance 


Note:  The  SE-CMM  project  intends  for  the  SAM  to  conform  to  the 

SEI's  common  appraisal  framework.  This  satisfies  a  higher  level 

project  requirement  that  the  SE-CMM  avoid  conflicts  with  the  CMM. 

The  individual  requirements  below  collectively  serve  this  purpose. 

6. 1  The  SE-CMM  is  the  reference  model  for  the  SAM. 

6.2  No  applicable  part  of  the  SE-CMM  is  excluded  in  the  SAM. 

6.3  SAM  uses  the  CAF  rating  scale; 

Satisfied 
Not  satisfied 
Not  applicable 
Not  rated 

6.4  Judgments  made  as  part  of  SAM  are  made  by  the  appraisal  team. 

6.5  Judgments  made  by  the  appraisal  team  address  base  practices, 
generic  practices,  and  process  areas. 

6.6  CAF  rules  of  evidence  will  be  applied  in  SAM;  i.e.,  data  will  be 
corroborated  by  multiple  sources.  (See  CAF  for  specific  rules  of 
evidence.) 

6.7  Confidence  ratings  will  be  determined  as  defined  in  the  CAF,  i.e., 
confidence  associated  with  both  the  criteria  selected  and  the 
execution  of  the  method  will  be  addressed. 

6.8  The  SAM  will  document  how  and  where  it  conforms  with  the  CAF. 

6.9  The  SAM  will  limit  the  number  of  appraisal  team  members  to  a 
minimum  of  four  and  a  maximum  of  eight. 

6.10  The  SAM  will  require  the  appraisal  team  to  document  the  domain 
of  the  appraisal  (project,  organization,  site). 


continued  on  next  page 
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Appendix  G:  Approved  SAM  Requirements,  continued 


7.0  Appraisal 
activities 


8.0  Confidence 

9.0  Tailoring 
10.0  Coverage 

11.0  Support 
materials 


The  following  activities  are  required  to  be  addressed  in  the  SAM,  to 
achieve  CAP  conformance: 

7.1  Planning 

7.2  Selection 

7.2. 1  Selection  of  appraised  entity 

7.2.2  Selection  of  appraisal  team 

7 . 3  Data  collection 

7 . 4  Data  consolidation 

7.5  Rating 

7.6  Reporting 

7.6  Post-appraisal  activities 


8.1  SAM  will  address  issues  related  to  confidence  and  risk  in  versions 
beyond  vl.O.  Version  1.0  of  SAM  does  not  meet  the  CAP  requirement 
to  address  confidence  and  risk. 


9. 1  SAM  will  describe  limits  of  tailoring  expected. 


10. 1  SAM  will  describe  coverage  requirements  related  to  a  particular 
confidence  rating. 


SAM  will  describe 

11.1  The  training  materials  required  for  an  SE-CMM  appraisal. 

1 1.2  The  supporting  briefing  materials  required  for  an  SE-CMM 

appraisal. 

1 1.3  The  supporting  data-gathering  materials  required  for  an  SE-CMM 

appraisal. 

1 1.4  Data-analysis  materials  required  for  an  SE-CMM  appraisal. 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF) 


SAM/CAF 

conformance 

matrix 


Requirement  in  CAF  1.0 

SAM  Paragraph 

Rl.  Method  documentation 
references: 

•  Reference  to  CMM  version 

Abstract 

1.1.2,  Assumptions 

•  Reference  to  CAF  version 

This  Appendix 

•  How  it  implements  CAF  appraisal 
activities 

1.1.1  Phases 

1.1.1  Relate  to  CB  A-IPI 

•  How  it  implements  CAF  appraisal 
artifacts 

1.1.3  Plan  Appraisal  Details 

1.1.4  Exit  Criteria 

2.2.5  Consolidate  Data  1 

2.2.6  Interview  Practitioners 

2.2.7  Consolidate  Data  2 

2.2.9  Review  Prelim  Findings 

2.2. 10  Develop  Draft  Rating 

2.2.13  Adjust  Draft  Rating 

2.2.14  Final  Briefing 

2.2.15  Sponsor  Briefing 

2.2. 16  Appraisal  Wrap-up 

•  How  it  implements  CAF  appraisal 
guidance 

This  Appendix 

R2.  Guidance  on: 

•  Planning  and  preparing  for 
appraisd 

2.1  Preparation 

♦  Conducting  appraisal 

2.1.4  to  2.3.3 

•  Reporting  results 

2.2.13  to  2.2.14  and 

2.3.1  to  2.3.2 

Table  H-1.  Traceability  Matrix  to  SEI  CAF 
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Appendix  H:  Traceability  Matrix  to  SEi  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R3.  Guidance  for: 

•  Identifying  appraisal  goals 

•  Identifying  appraisal  constraints 

•  Determining  suitability  wit 
goals/constraints/Abstract  to 

Scope 

2.1.1  Sponsor  Commit 

2.1.1  Sponsor  Commit 

2.1.2  Appraisal  Parameters 

2.1.3  Appraisal.  Details 

2.1.1  Sponsor  Commit 

R4.  Guidance  to  select  CMM  scope 

2.1.1  Purpose/Summary 
*  2. 1 . 1  T ailor  Parameters 

R5.  Guidance  to  select  organizational 
scope 

2. 1 .2  Summary  Description 

R6.  Guidance  to  obtain  organization, 
commitment 

2.1.1  to  2.1.3 

R7.  Appraisal  team  qualification 

criteria: 

•  >=  5  yrs  experience  for  majority 
of  team 

•  >=  25  yrs  experience  total  on  team 

•  >=  6  yrs  management  experience 
for  manager  on  team 

•  >=10  yrs  management  experience 
total  for  team 

*  1.3  Roles 

*  1.3  Roles 

*  1.3  Roles 

*  1.3  Roles 

R8.  Appraisal  team  leader  has 
experience:  ** 

•  Using  appraisal  method 

•  Managing  teams 

•  Facilitating  group  discussions 

•  Making  presentations 

*  1.1.3  Facilitator 

*  1.1.3  Roles 

1.1.3  Facilitator 

*  1.1.3  AT  leader 

R9.  Guidance  for  determining 
appropriate  team  size 

1.1.3  (1st  paragraph.) 

2.1.2  Table  2-5 
*  target  =  6  members 

Table  H-1.  Traceability  Matrix 

to  SEI  CAF,  continued 

continued  on  next  page 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

RIO.  Guidance  on  preparing  a  team  to 
do  appraisal 

1.4  Collect  data 

2.2.1  Table  2-10 

2.2.2  SAM  training 

R1 1.  Guidance  for  site  selection 

*2.1.2  Select  Parameters 
*  based  on  goals/project  status 

R12.  Guidance  for  project  selection 

*  2.1.2  Select  Parameters 

*  based  on  goals/project  status 

R13.  Guidance  for  participant 
selection 

1.1.3  Roles 

1.1.4  to  2.1.4 

R14.  Guidance  for  appraisal 
participants 

2.2.1 

R15.  Guidance  for  appraisal  planning; 

•  Identifies  appraisi  goals 

•  Identifies  appraisal  scope 

•  Identifies  appraisal  activities 

•  Provides  appraisal  schedule 

•  Identifies  resources 

•  Identifies  outputs  and  their  usage 

•  Identifies  anticipated  follow-on 
activities 

•  Documents  tailoring  and  trade-offs 

•  Identifies  risks  with  appraisal 
execution 

2.1.1  Sponsorship 

2.1.1  to  2.1.3 

2.1.3  Plan 

2.0  On-site 
(missing  figure) 

2.1.2  Summary 

2.1.3  Summary 

2.1.3  Summary 

2.3.0  Post-Appraisal 

*  2.3.4  Develop  Report 

1.2  Summary/Tailor 

*  2.1.3  Summary 

*  plan  includes  "force  field" 

R16.  Guidance  for  time  to  conduct 
appraisal 

2.1.0  Typical  Duration 

R17.  Guidance  for  appraisal  logistics 

2.1.3  Summary 
*  hours,  meals,  space,  etc. 
Appendix  C 

Table  H-1.  Traceability  Matrix 

to  SEI  CAF,  continued 

continued  on  next  page 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R18.  Define  artifacts  for: 

•  Recording  observations 

2.1.4  Questionnaire 

♦  Categorizing  observations 

2.2.3  Generate  Qs 

Appendix  C 

2.2.5  Data  Tracking  Sheet 

•  Classifying  observations 

2.2.3  Summary 

•  Validating  observations 

2.2.5  Update  DTS 

•  Recording  coverage 

2.2.6  Summary 

2.2.7  Update  DTS 

2.2.9  Review  Finds 

2.2.10  Update  DTS 

*  Use  PA  graphics  with 

•  Making  rating  decisions 

findings  stickies  to  display 
coverage 

2.2.7  Prelim  ratings 

2.2. 10  Draft  rating 

Reporting  findings  and  ratings 

2.2.12  Present  Finds 

2.2.14  Present  brief 

Managing  logistics 

2.2.15  Brief  sponsor 

2.3.2  Output  to  others 

2.1.3  Notes 

Appendix  G,  coord  checklist 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R19.  Guidance  to  implement  data 
collection  techniques: 

•  Administering  instruments 

2.1.4  Appraisal  participants 

•  Conducting  presentations 

2.2.1  Table  2-10 

2.2.9  Table  2-26 

2.2.12  Table  2-32 

*2.2.14  Table  2-36 
*  2.2.15  Table  2-38 

•  Conducting  interviews 

2.2.4  Table  2-16 

2.2.6  Table  2-20 

•  Reviewing  documentation 

*  2.2.3  Table  2-14 

*  suggestions  for  process 
documentation/artifacts  to  ask 
for  and  quality  attributes  (add 
appendix  template) 

R20.  Guidance  for  collecting  data: 

•  Extracting  data  from  data 

2.2.3  Table  2-14 

gathering  sessions 

*  needs  explicit  map  of 
instrument  answers  to  KPAs 

•  Recording  data  as  observations 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

•  Classifying  observations 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

•  Categorizing  as  CMM/non-CMM 

*  2.2.3  Table  2-14 

findings 

*  note  how  to  set  up  KPA 
graphics  with  non-CMM 
section(s) 

•  Categorizing  as  CMM/non-CMM 

*  2.2.8  Table  2-24 

findings 

*2.2.11  Table  2-30 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R21.  Guidance  for  validating 
observations; 

•  Corroboration  from  multiple. 

2.2.5  Table  2-18 

independent  sources 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

•  Interviews  by  doers  of  work  or 

2.2.6  interviews 

outcome  document  reviews 

R22.  Guidance  to  validate  a  portion  of 

2.1.4  Notes 

interview  data  by  KPA  (goal-related) 

2.2.3  Notes 

documentation 

2.2.4  Notes 

2.2.6  Notes 

R23.  Guidance  for  observation 
coverage  of  scope  and 
institutionalization: 

•  Each  goal  satisfied 

Not  applicable 

♦  Each  KPA  institutionalized 

*  2.1.4  Summary 

(common  features) 

*  How  generic  practices  are 
covered  in  Questionnaire  and 
document  review 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

♦  Each  ML  is  satisfied 

Not  applicable 

R24.  Mechanisms  to  adjust  collection 

2.2.3  Table  2-14 

to  obtain  coverage 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

R25.  Guidance  for  collected  data 

*  2.1.4  Summary 

traceability  to  outputs 

*  Add  graphic  display  of  +!- 
data  from  Questionnaire 

2.2.5  Table  2-18 

2.2.7  Table  2-22 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  Continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R26.  Require  ratings  of: 

*  KPA: 

•  Goals 

*  2.1.4  Summary 

*  How  generic  practices  are 
covered  in  Questionnaire  and 
document  review 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

Not  applicable 

R27.  If  appraisal  calculates  maturity 
level  rating,  consistent  w/  five  level 
scale  in  CMM  for  SW 

Not  applicable 

R28.  Rating  process  uses  the  rating 
values: 

•  Satisfied 

•  Unsatisfied 

•  Not  applicable 

•  Not  rated 

Not  applicable 

R29.  Rating  process  specifies: 

•  Goals  can  be  rated  when  coverage 
is  sufficient 

•  KPAs  can  be  rated  when  goals 
have  been 

•  Maturity  level  can  be  rated  when 
KPAs  have  been 

Not  applicable  (goals) 

Not  applicable  (goals) 

Not  applicable  (ML) 

R30.  Rating  process  uses  consensus 
of  team 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

R31.  Ratings  are  based  on  CMM  for 
software,  VI.  1 

Not  applicable 

R32.  Rating  process  requires  ratings 
to  be  based  on  findings 

2.2.10  Table  2-28 

2.2.13  Table  2-34 

R33.  Rating  process  specifies  goals 
are  rated: 

•  Satisfied  (conditions) 

•  Unsatisfied  (conditions) 

•  Not  applicable  (conditions) 

•  Not  rated  (conditions) 

Not  Applicable  (there  are  no 
goals  included  in  the  SE-CMM 
model) 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 
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Appendix  H;  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  continued 


SAM/CAF 
conformance 
matrix,  continued 


Requirement 

SAM  Paragraph 

R34.  Rating  process  for  KPAs  based 
on  goals 

Not  applicable 

R35.  Rating  process  which  specifies 
maturity  level  by  KPA  satisfaction 

Not  Applicable 

R36.  Reports  the  team  provides  are 
identified: 

2.1.1  Tailor  Parameters 

2.2.14  Summary 

2.3.1  Lessons  learned 

2.3.2  Report  to  others 

R37.  Reporting  includes  the  following 
data: 

•  Scope 

•  Selections  (site,  projects, 
participants,  team) 

•  Findings 

•  Ratings 

•  Risks  associated  with 
accuracy/completeness 

Appendix  B  (example) 

Appendix  B  (example) 

2.2.11  Table  2-30 

2.2.13  Table  2-34 

2.2. 10  Table  2-28 

2.2.13  Table  2-34 

*  22.2.10  Table  2-28 

*  Appendix  B  (example) 

R38.  Guidance  for  protecting 
confidentiality 

Appendix  Questionnaire 

2.2.1  Table  2-10 

Appendix  F  Training  Materials 
2.2.4  Table  2-16 

2.2.6  Table  2-20 

*  2.2.9  Notes 

*  add  note  to  remind  of 
confidentiality 

2.2.12  Table  2-32 

2.2.14  Summary 

2.2.15  Summary 

R39.  Guidance  for  retention  of 
records 

2.3.3 

Table  H-1.  Traceability  Matrix  to  SEI  CAF,  continued 
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Appendix  H:  Traceability  Matrix  to  SEI  CMM  Appraisal 
Framework  (CAF),  continued 


Table  H 


-1  Notes  *Items  preceded  by  are  suggestions  to  satisfy  this  requirement. 

There  is  an  implicit  assumption  which  should  be  explicit  (experience, 
"how  to"). 

**Item  where  SAM  deviates  acceptably  from  CAF  by  re-allocating 
requirement 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


A-71 
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Introduction 
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This  appendix  provides  the  references  for  documents  cited  within  the 
SAM. 
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Appendix  J:  SAM  Questionnaire 


Introduction 


In  this  appendix 


This  appendix  contains  the  instructions  and  forms  for  the  SAM 
questionnaire. 


The  following  table  provides  a  guide  to  the  information  found  in  this 
appendix. 


Topic 

See  Page 

Instructions  for  the  SAM  Questionnaire 

A-76 

Glossary 

A-78 

Respondent  Feedback 

A-81 

Site  Coordinator  Instructions  for  Distributing 
Questionnaire 

A-82 

Questionnaires  by  Process  Area 

A-87 

continued  on  next  page 
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Instructions  for  the  SAM  Questionnaire 


Purpose 


Respondent 

information 


Questionnaire 

structure 


The  purpose  of  this  questionnaire  is  to  gather  preliminary  data  on  your 
organization’s  systems  engineering  process  capability  for  the  upcoming 
systems  engineering  appraisal. 


Please  identify  yourself  and  your  project  or  team,  as  appropriate. 

Name: _  Date:  _ 

Project: _ _ 


Following  this  introduction  is  a  glossary  of  terms  used  in  the 
questionnaire. 

Following  the  glossary  is  a  place  for  you  to  provide  feedback  on  this 
questionnaire  to  the  developers  of  the  SE-CMM  appraisal  method. 
Rease  complete  this  form  after  you  have  completed  the  questionnaire. 
Your  comments  drive  improvements  to  the  appraisal  method  and 
questionnaire. 

The  body  of  the  questionnaire  has  three  pages  of  questions  for  each  of 
the  17  process  areas  of  the  Systems  Engineering  Capability  Maturity 
Model  (SE-CMM).  Each  process  area  begins  with  a  summary 
description  and  a  list  of  its  base  practices.  The  base  practices  are 
followed  by  three  series  of  questions  (parts  2,  3,  and  4),  each  series 
addressing  a  different  perspective  (performing  the  work,  managing  the 
process,  and  infrastructure  support). 


continued  on  next  page 
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Instructions  for  the  SAM  Questionnaire,  continued 


Instructions 


Before  You  Begin 

Please  read  the  glossary  to  become  familiar  with  how  specific  terms  are 
used  in  the  questionnaire.  Different  organizations  have  different  internal 
meanings  for  common  and  uncommon  terms.  The  glossary  provides  a 
context  for  you  to  understand  the  intended  meaning  of  the  terms 
throughout  your  appraisal.  Then  follow  these  steps  for  each  process 
area. 


Step 

Action 

1 

Read  the  process  area  summary. 

.  2 

Identify  the  base  practices  that  are  performed  on  your  project 
with  a  V  in  the  “Yes”  box.  Also  indicate  “No”  or  not 
applicable  (N/A).  (part  1) 

3 

If  you  had  no  “Yes”  answers,  please  proceed  directly  to  the 
next  process  area. 

4 

Answer  the  questions  that  follow  (parts  2, 3,  and  4)  from  the 
perspective  of  the  practices  for  which  you  answered  “Yes”  in 
part  1. 

Table  J-1.  Steps  for  Process  Areas. 

When  You  Are  Finished 

Please  fill  out  the  Respondent  Feedback  form.  Feel  free  to  comment  on 
both  content  and  format,  being  as  specific  as  possible.  Thank  you  for 
taking  the  time  to  fill  out  this  questionnaire. 
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Glossary 


Introduction 


Organizational 

terms 


This  glossary  defines  terms  used  in  the  questionnaire.  It  is 
recommended  that  you  read  through  theses  definitions  before  you  begin 
answering  the  questions  in  the  questionnaire. 


This  table  defines  how  the  terms  “organization”  and  “project”  are  used 
in  the  questionnaire. 


Term 

Definition 

organization 

In  the  context  of  the  SE-CMM,  “organization”  refers  to  the 
business  entity  being  appraised.  That  entity  should  have 
been  defined  for  you  by  those  who  gave  you  this 
questionnaire. 

Specifically,  an  organization  is  a  unit  within  a  company  or 
other  entity,  e.g.,  government  agency  or  branch  of  service, 
within  which  many  projects  are  managed  as  a  whole.  (All 
projects  within  an  organization  share  common  policies  at  the 
top  of  the  reporting  structure.) 

project 

In  the  context  of  the  SE-CMM,  a  project  is  an  entity  within 
an  organization  that  produces  system  engineering  work 
products,  which  are  typically  associated  with  a  particular 
deliverable  system. 

The  project  is  the  aggregate  of  effort  and  other  resources 
focused  on  developing  and/or  maintaining  a  specific 
product.  The  product  may  include  hardware,  software,  and 
other  components.  Typically  a  project  has  its  own  funding, 
cost  accounting,  and  delivery  schedule.  Some  projects  are 
organized  around  teaming  structures.  Consult  your  site 
coordinator  for  interpretation  of  project  if  your  experience 
appears  inconsistent  with  the  above  definition. 

Some  aspects  of  systems  engineering  process  are  associated 
with  the  project  and  some  with  the  organization.  Even 
though  you  may  primarily  work  in  one  of  these  contexts  or 
both,  answer  the  questions  based  on  your  best  knowledge  of 
what  is  happening  in  the  context  in  question. 

"organization" 

h — ^ 

organization- 
level  process 
issues 

T-J  r- 

— 

project-level 

"project" 

1 

process 

issues 

Table  J-2.  Organizational  Terms. 
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GlOSSdry,  Continued 


Process  terms  The  process  terms  in  the  table  below  are  arranged  in  the  order  that  they 
appear  in  the  questionnaire. 


Term 

Definition 

Process  area 

A  set  of  practices  (i.e.,  process  requirements)  that 
address  the  same  purpose. 

Process 

Any  specific  combination  of  machines,  tools,  methods, 
materials,  and/or  people  employed  to  attain  specific 
qualities  in  a  product  or  service. 

Work  products 

Anything  produced  by  a  process.  This  includes 
specifications,  documents,  engineering  drawings,  etc., 
not  just  the  product  delivered  to  the  ultimate  customer. 
Delivered  products  are  those  work  products  that  the 
customer  receives.  These  may  also  include 
specifications,  interim  documents,  prototypes,  etc.,  in 
addition  to  the  final  end  product  (the  deliverable 
system). 

Process 

performance 

data 

Measurements  that  are  used  to  manage  the  process  used 
on  your  project  or  in  your  organization.  These  are 
measures  of  the  actual  results  achieved  using  the 
process. 

This  process 

The  process  or  processes  that  your  project  or 
organization  uses  to  implement  the  process  area  about 
which  you  are  answering  questions. 

Organization's 

standard 

process 

(family  of 
processes) 

A  process  described  at  the  organizational  level  for  use 
by  projects  in  the  organization.  It  may  be  a  family  of 
processes  in  order  to  capture  the  different  classes  of 
processes  that  frequently  occur  in  organizations.  It  is 
intended  that  the  standard  process  be  tailored  into  a 
defined  process  to  meet  the  needs  of  specific  projects. 

Defect  review 

A  review  of  a  work  product,  interim  or  deliverable,  that 
occurs  prior  to  the  release  of  the  work  product  to  the 
next  process  step.  The  review  involves  the  creator  of 
the  product  and  subject  matter  peers  who  identify 
defects  in  the  product  that  would  make  it  unsuitable  for 
use  in  the  next  work  process.  It  is  a  form  of  static 
testing  of  the  work  product. 

Table  J-3.  Process  Terms. 
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Glossary,  continued 


Process  terms, 
continued 


Term 

Definition 

Process 

capability 

At  its  simplest,  process  capability  indicates  the  range  of 
results  expected  by  performing  a  process.  Process 
capability  implies  competence.  Initially  a  process  is 
chaotic.  (In  tMs  context,  chaotic  implies  a  high  degree 
of  variation.)  It  is  stable  when  special  causes  of 
variation  have  been  removed,  and  capable  when 
common  causes  of  variation  have  been  reduced  to  meet 
customer  validated  requirements  (specification  limits). 
This  is  the  process  improvement  framework  upon 
which  the  SE-CMM  process  dimension  is  based. 

Process  and 

product 

measures 

Note:  both  work  products  and  the  process  can  be 
measured. 

Product  measures  are  measurable  attributes  of  a 
product,  such  as  size  or  number  of  defects,  and 
generally  do  not  vary  over  time  (i.e.,  the  product 
measure  can  be  measured  at  any  time  and  get  the  same 
result). 

Process  measures  are  measurable  attributes  of  the 
process  used  to  produce  the  product,  such  as  resources 
expended  per  product  or  percent  rework.  (They  must 
be  measured  during  the  process;  after  the  process  has 
completed,  only  product  measures  may  be  taken.) 

Special  causes 
of  variation 

Special  causes  of  variation  are  assignable  to  people, 
places,  materials,  events,  etc.  They  are  causes  of 
variation  that  are  not  attributable  to  the  process  itself, 
although  they  may  be  attributable  to  some  aspect  of  its 
execution. 

Common 
causes  of 
variation 

Common  causes  are  causes  of  natural  variation  inherent 
in  a  process.  Removing  common  causes  of  variation 
involves  making  changes  to  the  process  itself. 

Table  J-3.  Process  Terms,  continued 
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Respondent  Feedback 


Respondent 

information 


Feedback 


Please  identify  yourself  and  your  project. 

Name: _  Date:  _ 

Organization: _ 

Amount  of  time  spent  filling  out  questionnaire: _ 

PAs  you  were  given  to  answer  questions  about:  _ all 

If  not  all,  then  please  list  the  numbers  you  were  given: _ 


We  would  greatly  appreciate  your  comments  on  the  questionnaire.  The 
developers  of  the  SE-CMM  appraisal  method  will  use  this  feedback  to 
improve  the  SE-CMM,  the  appraisal  method,  and  the  questionnaire. 
Feel  free  to  comment  on  both  content  and  format;  the  more  specific,  the 
better.  We  are  specifically  interested  in  the  following  areas:  clarity  of 
instructions,  usefulness  of  the  glossary,  other  terms  that  need  to  be 
defined,  structure  of  the  questionnaire,  and  understandability  of  the 
questions.  Thank  you  for  your  time  and  input! 
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Site  Coordinator  Instructions  for  Distributing  Questionnaire 


Introduction 


Time  constraints 


Recommended 

administration 

approach 


The  SE-CMM  questionnaire  is  the  first  step  in  a  data  gathering  process 
designed  to  provide  the  managers  and  practitioners  with  insight  into  the 
organization's  systems  engineering  practices.  When  determining  who 
should  fill  out  the  questionnaire,  it  is  important  to  choose  individuals 
who  will  provide  answers  that  represent  the  entire  project/organization. 


It  typically  takes  around  two  hours  for  an  experienced  project  lead-level 
systems  engineer  to  complete  the  entire  SE-CMM  questionnaire.  For  a 
subject  matter  expert  to  fill  out  a  single  questionnaire  typically  takes 
between  5-10  minutes.  Questionnaire  response  time  is  reduced  when 
the  respondents  have  immediate  access  to  a  site  coordinator  who 
understands  the  model/appraisal  method,  and  when  the  entire 
questionnaire  is  answered  in  one  sitting. 


To  maximize  the  use  of  both  the  site  coordinator's  and  respondents' 
time,  it  is  recommended  that  one  or  two  "appointments"  be  set  up  in  a 
room  large  enough  to  accommodate  all  respondents  with  a  suitable 
writing  surface,  e.g.,  a  large  table  or  several  small  tables.  Respondents 
are  invited  to  schedule  themselves  for  one  of  the  two  appointments. 
(One  is  optimal,  because  then  all  respondents  hear  answers  to  questions 
together.  However,  at  many  sites  getting  the  individuals  needed  to 
answer  the  questions  at  the  same  meeting  is  often  difficult.)  Before  the 
individuals  start  filling  out  the  questionnaires,  the  site  coordinator 
introduces  the  respondents  to  the  model  and  appraisal,  distributes  the 
questionnaire,  and  makes  clear  that  the  answers  are  to  reflect  their 
individual  opinions.  He/she  also  makes  clear  that  the  site  coordinator  is 
there  to  clarify  terminology  and  concepts,  and  respondents  are 
encouraged  to  voice  questions.  The  site  coordinator  can  then  answer  to 
the  entire  group  assembled.  The  site  coordinator  also  records  any 
questions  he/she  cannot  answer  so  he/she  can  contact  an  SE-CMM 
facilitator  for  guidance. 

After  each  respondent  is  finished,  the  site  coordinator  collects  the 
questionnaire  and,  if  not  already  filled  in  by  respondent,  completes  the 
'time  spent'  portion  of  the  feedback  form. 
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Site  Coordinator  Distribution  Instructions,  Continued 


Questionnaire  To  maximize  the  accuracy  of  initial  responses  to  the  questions  in  the  SE- 
distribution  table  CMM  questionnaire,  it  is  recommended  that  the  questionnaires  be 
distributed  to  individuals  with  the  skills  and  roles  expressed  in  the 
following  table. 


Process  Area 

Primary 

Respondents 

Secondary 

Respondents 

01:  Analyze 

Candidate  Solutions 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Any  senior  practitioner 
with  significant 
system  design 
experience 

02:  Derive  and 
Allocate  Requirements 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

03:  Develop  Physical 
Architecture 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

04:  Integrate 
Disciplines 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Senior  specialty 
engineers  (e.g., 
reliability,  safety, 
manufacturing,  human 
factors)  working  on 
the  projects  selected 
for  appraisal 

05:  Integrate  System 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

06:  Understand 
Customer  Needs  and 
Expectations 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

•  Technical  marketing 
personnel 

•  Proposal  personnel 

•  Customer  service 
personnel 

07:  Verify  and 

Validate  System 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

System  verification 
manager  or  senior  test 
engineers 

Table  J-4.  Questionnaire  Distribution  Table. 
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Site  Coordinator  Distribution  Instructions,  continued 


Questionnaire 
distribution  table, 
continued 


Process  Area 

Primary 

Respondents 

Secondary 

Respondents 

08:  Ensure  Quality 

Senior  project-level 
quality  manager  or 
lead  (in  environments 
with  shared  quality 
leadership 

responsibility,  systems 
engineering  lead  for 
the  project) 

•  Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

•  Organizational 
quality  manager, 
total  quality 
management 
coordinator 

09:  Manage 
Configurations 

Senior  project-level 

CM  manager  for  the 
projects  selected  for 
appraisal 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

10:  Monitor  and 
Control  Technical 
Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

11:  Plan  Technical 
Effort 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

12:  Manage  Risk 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Project  or  program 
manager  for  the 
projects  selected  for 
appraisal 

13:  Define 
Organization's 

Systems  Engineering 
Process 

Individuals 
responsible  for 
defining  organization 
level  processes;  may 
be  part  of  the  quality 
leadership  area, 
policies/procedures 
area,  or  other  support 
group 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Table  J-4.  Questionnaire  Distribution  Table,  continued 
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Site  Coordinator  Distribution  Instructions,  Continued 


Questionnaire 
distribution  table, 
continued 


Process  Area 

Primary 

Respondents 

Secondary 

Respondents 

14:  Improve 
Organization's 

Systems  Engineering 
Processes 

Individuals 
responsible  for 
deploying  organization 
level  process 
improvement  activities; 
may  be  part  of  the 
quality  leadership  area, 
policies/procedures 
area,  or  other  support 
group 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

15:  Manage  Product 
Evolution 

Individuals  at 
organization  level 
responsible  for 
strategic  product  line 
positioning  and 
advancement;  may  be 
in  R&D,  technical 
marketing,  or  other 
support  structure 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

16:  Manage  Systems 
Engineering  Support 
Environment 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Individuals  at 
organization  level 
involved  in  deploying 
new  development 
technologies 

17:  Manage  Systems 
Engineering  Training 

Individuals 
responsible  for 
planning, 
development,  and 
deployment  of 
organization-level 
training;  may  be  part 
of  an  R&D  group, 
training  department,  or 
other  support  structure 

Systems  engineering 
leads  for  the  projects 
selected  for  appraisal 

Table  J-4.  Questionnaire  Distribution  Table,  continued 
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Site  Coordinator  Distribution  Instructions,  Continued 


Distribution 

recommendation 


Even  though  there  are  other  roles  called  out  as  primary  targets  for  the 
questionnaire  in  certain  instances,  having  the  SE  leads  answer  all  the 
PAs  provides  an  overall  context  of  how  things  appear  from  the  project 
viewpoint,  which  can  be  very  valuable.  Therefore,  it  is  recommended 
that  the  SE  leads  complete  all  the  PAs  whenever  feasible. 
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PA  01 :  Analyze  Candidate  Solutions 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  purpose  of  Analyze  Candidate  Solutions  is  to  perform  studies  and  analyses  that 
result  in  the  selection  of  a  solution  to  meet  the  specified  constraints  of  the  situation 
that  generated  the  need  for  analysis.  Analyze  Candidate  Solutions  involves  defining 
the  approach  and  evaluation  criteria  for  the  analysis,  as  well  as  for  choosing, 
selecting,  and  studying  the  candidate  solutions.  Communication  of  the  rationale 
and  results  of  the  analysis  must  also  be  accomplished. 


The  following  list  contains  base  practices  that  are  essential  elements  of  good 
systems  engineering: 


Yes  No 

[]  [] 

[]  [] 

[]  [] 

[]  [] 
[]  [] 
[]  [] 


N/A 

[  ]  Establish  evaluation  criteria  based  on  the  identified  problenti  and  its  defined 
constraints. 

[  1  Define  the  general  approach  for  the  analysis,  based  on  the  established  evaluation 
criteria. 

[  ]  Identify  alternatives  for  evaluation  in  addition  to  those  provided  with  the  problem 
statement. 

[  ]  Analyze  the  competing  candidate  solutions  against  the  established  evaluation  criteria. 

[  ]  Select  the  solution  that  satisfies  the  established  evaluation  criteria. 

[  ]  Capture  the  disposition  of  each  alternative  under  consideration  and  the  rationale  for 
the  disposition. 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  01 :  Analyze  Candidate  Solutions,  Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5-2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  01:  Analyze  Candidate  Solutions,  Continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization’s 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  01:  Analyze  Candidate  Solutions 
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PA  02:  Derive  and  Allocate  Requirements 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  the  Derive  and  Allocate  Requirements  is  to  analyze  the  system  and 
other  requirements  and  derive  a  more  detailed  and  precise  set  of  requirements. 
These  derived  requirements  are  allocated  to  system  functions,  people,  and 
supporting  processes,  products,  and  services,  which  can  be  used  to  synthesize 
solutions.  This  process  area  addresses  both  the  analysis  of  system-level 
requirements  and  the  allocation  of  system-level  or  derived  requirements  to  lower 
level  functions.  This  analysis  involves  addressing  the  concept  of  operations, 
functional  partitioning,  and  performance  allocation,  as  well  as  capturing  the  status 
and  traceability  of  requirements. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
"  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 

[]  [] 

[]  [] 

[]  [] 
[]  [] 
[]  [] 
[]  [] 
[]  [] 
[]  [] 


N/A 

[  ]  Develop  a  detailed  operational  concept  of  the  interaction  of  the  system,  the  user,  and 
the  environment,  that  satisfies  the  operational  need. 

[  ]  Identify  key  requirements  that  have  a  strong  influence  on  cost,  schedule, 
functionality,  or  performance. 

[  ]  Partition  requirements  into  groups  of  requirements  based  on  established  criteria,  such 
as  similar  functionality,  performance,  or  coupling,  to  facilitate  and  focus  the 
requirements  analysis. 

[  ]  Derive,  from  the  system  and  other  (e.g.,  environmental)  requirements,  requirements 
that  may  be  logically  inferred  and  implied  as  essential  to  system  effectiveness. 

[  ]  Identify  the  requirements  associated  with  external  interfaces  to  the  system  and 
interfaces  between  functional  partitions. 

[  ]  Allocate  requirements  to  functional  partitions,  system  elements,  people,  and  support 
elements  to  support  synthesis  of  solutions. 

[  ]  Analyze  requirements  to  ensure  that  they  are  verifiable  by  the  methods  available  to 
the  development  effort. 

[  ]  Maintain  requirements  traceability  to  ensure  that  lower  level  (derived)  requirements 
are  necessary  and  sufficient  to  meet  the  objectives  of  higher  level  requirements. 

[  ]  Capture  system  and  other  requirements,  derived  requirements,  derivation  rationale, 
allocations,  traceability,  and  requirements  status. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  02:  Derive  and  Allocate  Requirements,  continued 


2 .  Performing  Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area. 

Comments: 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  02:  Derive  and  Allocate  Requirements,  Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  02:  Derive  and  Allocate  Requirements,  Continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  02:  Derive  and  Allocate  Requirements 
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PA  03:  Develop  Physical  Architecture 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Develop  Physical  Architecture  is  to  transform  the  functional 
architecture,  as  defined  by  the  Derive  and  Allocate  Requirements  process  area,  into 
the  physical  architecture  for  the  system.  It  involves  deriving  the  physical 
architecture  requirements,  identifying  the  key  design  issues,  determining  the 
physical  structure  and  interfaces,  and  allocating  the  physical  architecture 
requirements.  The  practices  described  herein  are  expected  to  be  performed 
iteratively  until  the  design  is  handed  off  to  the  implementing  or  component 
engineering  disciplines. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
-  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 

[]  [] 

[]  [] 

[]  [] 
[]  [] 


[]  [] 


N/A 

[  ]  Derive  the  requirements  for  the  physical  architecture. 

[  ]  Identify  the  key  design  issues  that  must  be  resolved  to  support  successful 
development  of  the  system. 

[  ]  Generate  physical  structure  alternative(s)  and  constraints,  and  select  a  solution  in 
accordance  with  the  Analyze  Candidate  Solutions  process  area, 

[  ]  Develop  the  physical  architecture's  interface  requirements  for  the  chosen  physical 
structure. 

[  ]  Allocate  the  physical  architecture  requirements  to  the  chosen  physical  structure. 

[  ]  Maintain  requirements  traceability  for  the  physical  architecture  requirements  to  ensure 
that  lower  level  (derived)  requirements  are  necessary  and  sufficient  to  meet  the  needs 
of  higher  level  requirements  or  design. 

[  ]  Describe  the  physical  architecture  by  capturing  the  design  results  and  rationale. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  ”Yes”.  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  03:  Develop  Physical  Architecture,  Continued 


2 .  Performing  Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area. 

Comments: 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  03:  Develop  Physical  Architecture,  Continued 


3 .  Process 
management 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 
Solutions  process  area  is  managed  using  the  following  generic  practices. 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization’s  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  03:  Develop  Physical  Architecture,  Continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  03:  Develop  Physical  Architecture 
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PA  04:  Integrate  Disciplines 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Integrate  Disciplines  is  to  identify  those  disciplines  necessary  for 
effective  system  development  and  create  an  environment  in  which  they  jointly  and 
effectively  work  together  toward  a  common  agenda.  Each  discipline’s  unique 
expertise  and  concerns  are  forwarded  and  considered,  but  the  focus  on  total  system 
development  is  maintained.  These  disciplines  may  include,  but  are  not  limited  to, 
marketing,  manufacturing,  component  design,  development  (e.g.,  hardware, 
software,  or  firmware),  reliability,  maintainability,  supportability,  human  factors, 
logistics,  safety,  and  security.  It  is  critical  to  be  able  to  meld  such  disciplines 
without  sacrificing  their  parochial  interests  concerning  issues  important  to  and 
unique  to  each  discipline.  This  environment  must  persist  throughout  the  system 
development  life  cycle. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
"  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 

[]  [] 
[]  [] 
[]  [] 
[]  [] 

[]  [] 
[]  [] 


N/A 

[  ]  Identify  the  disciplines  that  are  directly  or  indirectly  essential  to  system  development. 

[  ]  Familiarize  individuals  involved  in  the  development  effort  with  the  various 
disciplines  and  their  roles  in  creating  a  successful  system. 

[  ]  Actively  promote  cross-discipline  understanding  within  the  development  team. 

[  ]  Establish  methods  for  interdisciplinary  coordination. 

[  ]  Establish  methods  for  identifying  and  resolving  interdisciplinary  issues. 

[  ]  Follow  established  interdisciplinary  methods  to  achieve  integrated  solutions  to 
identified  issues  or  problems. 

[  ]  Communicate  results  of  interdisciplinary  activities  to  affected  groups. 

[  ]  Develop  project  goals  and  ensure  that  each  project  member  and  direct  support  person 
is  fully  aware  of  them. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  04:  Integrate  Disciplines,  Continued 


2 .  Performing 
the  work 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2,2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  04:  integrate  Disciplines,  Continued 


3 .  Process 
management 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 
Solutions  process  area  is  managed  using  the  following  generic  practices. 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization’s  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  04:  Integrate  Disciplines,  Continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  04:  Integrate  Disciplines 
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PA  05:  Integrate  System 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Integrate  System  is  to  ensure  that  system  elements  will  function  as  a 
whole.  This  primarily  involves  identifying,  defining,  and  controlling  interfaces,  as 
well  as  verifying  system  functions  that  require  multiple  system  elements.  The 
activities  associated  with  Integrate  System  occur  throughout  the  entire  life  cycle  of 
system  development. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it’s  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 

[]  [] 

[]  [] 

[]  [] 

[]  [] 

[]  [] 

[]  [] 


N/A 

[  ]  Develop  detailed  descriptions  of  the  interfaces  implied  by  the  system  architecture. 

[  ]  Communicate  the  interface  definitions  and  coordinate  change  requests  with  all  system 
element  developers  who  could  be  affected  by  interface  changes. 

[  ]  Verify  the  receipt  of  each  system  element  required  to  assemble  the  system  in 
accordance  with  the  physical  architecture, 

[  ]  Verify  the  implemented  design  features  of  developed  or  purchased  system  elements 
against  their  requirements. 

[  ]  Verify  that  the  system  element  interfaces  comply  with  the  interface  requirements 
prior  to  assembly. 

[  ]  Assemble  aggregates  of  system  elements  in  accordance  with  the  established 
integration  strategy, 

[  ]  Check  the  integrated  system  interfaces  in  accordance  with  the  established  integration 
strategy. 

[  ]  Develop  an  integration  strategy  and  supporting  documentation  which  identifies  the 
optimal  sequence  for  receipt,  assembly,  and  activation  of  the  various  components  that 
make  up  the  physical  architecture  of  the  system. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  05l  IntOQrstG  SystGin,  continued 


2 .  Performing 
the  work 

Comments: 


In^cate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  r)roceHiirp<j9 

r 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area‘s 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  05:  Integrate  System,  continued 


3 .  Process 
management 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 
Solutions  process  area  is  managed  using  the  following  generic  practices. 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1  ^ 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  05:  Integrate  System,  continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3,1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization’s 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  05:  Integrate  System 
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PA  06:  Understand  Customer  Needs  and  Expectations 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  purpose  of  Understand  Customer  Needs  and  Expectations  is  to  elicit,  stimulate, 
analyze,  and  communicate  customer  needs  and  expectations  to  obtain  a  better 
understanding  of  what  will  satisfy  the  customer.  Understand  Customer  Needs  and 
Expectations  involves  engaging  the  customer  or  their  surrogate  in  ongoing  dialogue 
designed  to  translate  his/her  needs  and  expectations  into  a  verifiable  set  of 
requirements  which  the  customer  understands  and  which  provide  the  basis  for 
agreements  between  the  customer  and  the  systems  engineering  effort. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 

[]  [] 
[]  [] 

[]  [] 


N/A 

[  ]  Elicit  customer  needs,  expectations,  and  measures  of  effectiveness. 

[  ]  Analyze  customer  needs  and  expectations  to  develop  a  preliminary  operational 
concept  of  the  system  as  appropriate. 

[  ]  Develop  a  statement  of  system  requirements. 

[  ]  Obtain  concurrence  from  the  customer  that  the  agreed  upon  customer  requirements 
satisfy  their  needs  and  expectations. 

[  ]  Inform  the  customer  on  a  regular  basis  about  the  status  and  disposition  of  needs, 
expectations,  and  measures  of  effectiveness. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 

Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  06:  Understand  Customer  Needs  and  Expectations,  continued 


3 .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization’s  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  06:  Understand  Customer  Needs  and  Expectations,  Continued 


4 .  Supporting 
infrastructure 

Comments: 

ID  Are  the  following  visible  and  available  to  those  using  the  V 

organization's  processes? 

3.1.1  Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1  Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1  Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 


end  of  PA  06:  Understand  Customer  Needs  and  Expectations 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 
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PA  07:  Verify  and  Validate  System 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Verify  and  Validate  System  is  to  ensure  that  the  developer/supplier 
team  performs  increasingly  comprehensive  evaluations  to  ensure  that  evolving  work 
products  will  meet  all  requirements.  The  activities  associated  with  Verify  and 
Validate  System  begin  early  in  the  development,  address  all  work  products 
(including  requirements  and  design),  and  continue  through  system  element 
development  and  integration.  The  scope  of  verification  covers  development  of  the 
full  system,  as  well  as  its  production,  operation  and  support.  Validation  involves 
evaluation  of  the  customer  requirements  against  customer  needs  and  expectations, 
and  evaluation  of  the  delivered  system  to  meet  the  customer's  operational  need  in 
the  most  representative  environment  achievable. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
”  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[1  11 

[1  [] 

11  □ 
11  [] 
[]  [] 
[1  11 


N/A 

[  ]  Establish  plans  for  verification  and  validation  that  identify  the  overall  requirements, 
objectives,  resources,  facilities,  special  equipment,  and  schedule  applicable  to  the 
system  development. 

[  ]  Define  the  methods,  process,  reviews,  inspections  and  tests  by  which  incremental 
products  are  verified  against  established  criteria  or  requirements  established  in  a 
previous  phase. 

[  ]  Define  the  methods,  process,  and  evaluation  criteria  by  which  the  system  or  product 
is  verified  against  the  system  or  product  requirements. 

[  1  Define  the  methods,  process,  and  evaluation  criteria  by  which  the  system  or  product 
will  be  validated  against  the  customer’s  needs  and  expectations. 

[  ]  Perform  the  verification  and  validation  activities  that  are  specified  by  the  verification 
and  validation  plans  and  procedures,  and  capture  the  results. 

[  ]  Compare  the  collected  test,  inspection,  or  review  results  with  established  evaluation 
criteria  to  assess  the  degree  of  success. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  07:  Verify  &  Validate  System,  continued 


2 .  Performing 
the  work 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  07:  Verify  &  Validate  System,  continued 

3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area?’ 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

^  Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  07:  Verify  &  Validate  System,  continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3,1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization’s 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  07: 
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PA  08:  Ensure  Quality 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Ensure  Quality  is  to  address  not  only  the  quality  of  the  system,  but 
also  the  quality  of  the  process  being  used  to  create  the  system  and  the  degree  to 
which  the  project  follows  the  defined  process.  The  underlying  concept  of  this 
process  area  is  that  high  quality  systems  can  only  be  produced  on  a  continuous 
basis  if  a  process  exists  to  continuously  measure  and  improve  quality,  and  this 
process  is  adhered  to  rigorously.  Key  aspects  of  the  process  required  to  develop 
high  quality  systems  are  measurement,  analysis,  and  corrective  action. 

This  is  not  meant  to  imply  that  those  managing  and/or  assuring  the  quality  of  work 
products  and  processes  are  solely  responsiWe  for  the  quality  of  the  work  product 
outputs.  On  the  contrary,  the  primary  responsibility  for  "building  in"  quality  lies 
with  the  builders.  The  support  of  a  quality  management  process  adds  confidence 
for  the  developers,  management,  and  customers  that  all  aspects  of  quality 
management  are  seriously  considered  and  acted  upon  by  the  organization  and 
reflected  in  its  products. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
"  it’s  enough  that  it  is  known  who  performs  it. 


Yes  No 
[1  [1 

[]  [] 
[]  [] 
[]  [] 

[1  [] 

[1  [] 


N/A 

[  ]  Ensure  the  defined  system  engineering  process  is  adhered  to  during  the  system 
development  life  cycle. 

[  ]  Evaluate  work  product  measures  against  the  requirements  for  work  product  quality. 

[  ]  Measure  the  quality  of  the  systems  engineering  process  used  by  the  project. 

[  ]  Analyze  quality  measurements  to  develop  recommendations  for  quality  improvement 
or  corrective  action  as  appropriate. 

[  ]  Promote  atmosphere  that  encourages  employees  to  be  attentive  to  quality  issues  and 
report  quality  problems. 

[  ]  Initiate  activities  that  address  identified  quality  issues  or  quality  improvement 
opportunities. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  08:  Ensure  Quality,  Continued 


2 .  Performing 
the  work 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  08:  Ensure  Quality,  Continued 


3.  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2,1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization’s  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  08:  Ensure  Quality,  Continued 


4  .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

_ 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  08:  Ensure  Quality 
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PA  09:  Manage  Configurations 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  purpose  of  Manage  Configurations  is  to  maintain  data  and  status  of  identified 
configuration  units,  and  to  analyze  and  control  changes  to  the  system  and  its 
configuration  units.  Managing  the  system  configuration  involves  providing 
accurate  and  current  configuration  data  and  status  to  developers  and  customers. 

This  process  area  is  applicable  to  all  work  products  that  are  desired  to  be  placed 
under  configuration  management.  An  example  set  of  work  products  that  may  be 
placed  under  configuration  management  could  include  hardware  and  software 
configuration  items,  design  rationale,  requirements,  product  data  files,  or  trade 
studies. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[1  [] 
[]  [] 
[]  [] 
[]  [] 
[]  [] 


N/A 

[  ]  Decide  among  candidate  methods  for  configuration  management 
[  ]  Identify  configuration  units  that  constitute  identified  baselines. 

[  ]  Maintain  a  repository  of  configuration  data. 

[  ]  Control  changes  to  established  configuration  units. 

[  ]  Communicate  changes  to  status,  proposed  changes,  and  configuration  data  to  affected 
groups. 


If  all  of  your  answers  were  ”No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  ”Yes'\  Feel  free  to  add  clarifying  comments  in  the  margin. 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


A-117 


PA  09:  Manage  Configurations,  Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  09:  Manage  Configurations,  continued 


4 ,  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization’s 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  09:  Manage  Configurations 
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PA  10:  Manage  Risk 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  purpose  of  Manage  Risk  is  to  identify,  assess,  monitor,  and  mitigate  risks  to 
the  success  of  both  the  systems  engineering  activities  and  the  overall  technical 
effort.  This  process  area  continues  throughout  the  life  of  the  project.  Similar  to 
Plan  Technical  Effort  and  Monitor  and  Control  Technical  Effort  process  areas,  the 
scope  of  this  process  area  includes  both  the  systems  engineering  activities  and  the 
overall  technical  project  effort,  as  the  systems  engineering  effort  on  the  project 
cannot  be  considered  successful  unless  the  overall  technical  effort  is  successful. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
"  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 


[]  [] 

[]  [] 

[]  [] 
[]  [] 
[]  [] 


N/A 

[  ]  Develop  a  plan  for  risk  management  activities  that  is  the  basis  for  the  risk 

identification,  assessment,  mitigation,  and  monitoring  activities  for  the  life  of  the 
project. 

[  ]  Identify  project  risks  by  examining  project  objectives  with  respect  to  the  alternatives 
and  constraints  and  identifying  what  can  go  wrong. 

[  ]  Assess  risks  and  determine  the  probability  of  occurrence  and  consequence  of 
realization. 

[  ]  Obtain  formal  recognition  of  the  project  risk  assessment. 

[  ]  Implement  the  risk  mitigation  activities. 

[  ]  Monitor  risk  mitigation  activities  to  ensure  the  desired  results  are  being  obtained. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes”.  Feel  free  to  add  clarifying  comments  in  the  margin. 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2,1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3,2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  10:  Manage  Risk,  continued 


3 .  Process 
management 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 
Solutions  process  area  is  managed  using  the  following  generic  practices. 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2,3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


A-121 


PA  10r  Mand96  Risk,  Continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
an^yzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4,1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  10:  Manage  Risk 
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PA  11:  Monitor  and  Control  Technical  Effort 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  purpose  of  Monitor  and  Control  Technical  Effort  is  to  provide  adequate 
visibility  of  actual  progress  and  risks.  Visibility  encourages  timely  corrective  action 
when  performance  deviates  significantly  from  plans. 

Monitor  and  Control  Technical  Effort  involves  directing,  tracking  and  reviewing  the 
project's  accomplishments,  results,  and  risks  against  its  documented  estimates, 
commitments,  and  plans.  A  documented  plan  is  used  as  the  basis  for  tracking  the 
activities  and  risks,  communicating  status,  and  revising  plans. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 
[]  [] 
[]  [] 
[]  [] 

[]  [] 


N/A 

[  ]  Direct  technical  effort  in  accordance  with  technical  management  plans. 

[  ]  Track  actual  resource  utilization  against  technical  management  plans. 

[  ]  Track  performance  against  the  established  technical  parameters. 

[  ]  Review  performance  against  the  technical  management  plans. 

[  ]  Analyze  issues  resulting  from  technical  parameter  tracking  and  review  activities  to 
determine  corrective  actions. 

[  ]  Take  corrective  actions  when  actual  results  deviate  significantly  from  plans. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  1 1 :  Monitor  and  Control  Technical  Effort,  Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

< 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  11:  Monitor  and  Control  Technical  Effort,  continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

3,1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1,1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  1 1 :  Monitor  and  Control  Technical  Effort 
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PA  12:  Plan  Technical  Effort 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Plan  Technical  Effort  is  to  establish  plans  that  provide  the  basis  for 
scheduling,  costing,  controlling,  tracking  and  negotiating  the  nature  and  scope  of 
the  technical  work  involved  in  the  system  development.  System  engineering 
activities  must  be  integrated  into  comprehensive  technical  planning  for  the  entire 
project. 

Plan  Technical  Effort  involves  developing  estimates  for  the  work  to  be  performed, 
obtaining  necessary  commitments  from  interfacing  groups,  and  defining  the  plan  to 
perform  the  work. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 

[]  [] 
[]  [] 
[]  [] 
[]  [] 

[]  [] 
[]  [] 
[]  [] 


[]  [] 
[]  [] 


N/A 

[  ]  Identify  resources  critical  to  the  technical  success  of  the  project. 

[  ]  Develop  estimates  for  the  factors  that  affect  the  magnitude  and  technical  feasibility  of 
the  project. 

[  ]  Develop  cost  estimates  for  all  technical  resources  required  by  the  project. 

[  ]  Determine  the  technical  process  to  be  used  on  the  project. 

[  ]  Identify  technical  activities  for  the  entire  life  cycle  of  the  project. 

[  ]  Define  specific  processes  to  support  effective  interaction  with  the  customer(s)  and 
supplier(s). 

[  ]  Develop  technical  schedules  for  the  entire  project  life  cycle. 

[  ]  Establish  technical  parameters  with  thresholds  for  the  project  and  the  system. 

[  ]  Use  the  information  gathered  in  planning  activities  to  develop  technical  management 
plans  that  will  serve  as  the  basis  for  tracking  the  salient  aspects  of  the  project  and  the 
systems  engineering  effort. 

[  ]  Review  the  technical  management  plans  with  all  affected  groups  and  individuals. 

[  ]  Obtain  commitment  to  the  technical  management  plans  from  all  affected  groups  and 
individuals. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  12:  Plan  Technical  Effort,  Continued 


2  .  Performing  Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area. 

Comments: 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  12:  Plan  Technical  Effort,  Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

i  Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 


A-128 


SECMM-94-06ICMU/SEI-94-HB-05  v1 .0 


PA  12:  Plan  Technical  Effort,  Continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  12:  Plan  Technical  Effort 
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PA  13:  Define  Organization's  Systems  Engineering  Process 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Define  Organization's  Systems  Engineering  Process  is  to  create  and 
manage  the  organization's  standard  systems  engineering  processes,  which  can 
subsequently  be  tailored  by  a  project  to  form  the  unique  processes  that  it  will  follow 
in  developing  its  systems  or  products. 

Define  Organization's  Systems  Engineering  Process  involves  defining  the  process 
that  will  meet  the  business  goals  of  the  organization,  as  well  as  designing, 
developing  and  documenting  organizational  process  assets  which  are  collected  and 
maintained.  Process  assets  is  a  term  used  to  emphasize  the  investment  nature  of 
defining  organizational  processes;  assets  include  example  processes,  process 
fragments,  process-related  documentation,  process  arcWtectures,  process  tailoring 
rules  and  tools,  and  process  measurements. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
"  it's  enough  that  it  is  known  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Establish  goals  for  the  organization’s  systems  engineering  process  from  the 

organization's  business  goals. 

[  ]  [  ]  [  ]  Collect  and  maintain  systems  engineering  process  assets. 

[  ]  [  ]  [  ]  Develop  the  organization's  standard  systems  engineering  process. 

[  ]  [  ]  [  ]  Define  guidelines  for  tailoring  the  organization's  standard  systems  engineering 

process  for  project  use  in  developing  the  project's  defined  process. 

If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  13:  Define  Organization's  Systems  Engineering  Process, 

Continued 


2 .  Performing  Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area. 

Comments: 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate?  i 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

!  Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  13:  Define  Organization's  Systems  Engineering  Process, 

Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  13:  Define  Organization's  Systems  Engineering  Process, 

Continued 


4  .  Supporting  Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
infrastructure  analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
^  visible  to  the  organization  as  a  whole. 

Comments: 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  13:  Define  Organization's  Systems  Engineering  Process 
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PA  14:  Improve  Organization's  Systems  Engineering  Processes 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  pu^ose  of  Improve  Organization's  System  Engineering  Processes  is  to  gain 
competitive  advantage  by  continuously  improving  the  effectiveness  and  efficiency 
of  the  systems  engineering  processes  used  by  the  organization.  It  involves 
developing  an  understanding  of  the  organization's  processes  in  the  context  of  the 
organization's  business  goals,  analyzing  the  performance  of  the  processes,  and 
explicitly  planning  and  deploying  improvements  to  those  processes. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 

you  do  not  have  to  personally  be  involved  in  performing  the  practice 

”  it's  enough  that  it  is  known  who  performs  it. 

Yes  No  N/A 

[  ]  [  ]  [  ]  Appraise  the  existing  processes  being  performed  in  the  organization  to  understand 

their  strengths  and  weaknesses. 

[  ]  [  ]  [  ]  Plan  improvements  to  the  organization’s  processes  based  on  an  analysis  of  the  impact 
of  potential  improvements  on  achieving  the  goals  of  the  processes. 

[  ]  [  ]  [  ]  Change  the  organization’s  standard  systems  engineering  process  to  reflect  targeted 

improvements. 

[  ]  [  ]  [  ]  Communicate  process  improvements  to  existing  projects  and  to  other  affected 

groups,  as  appropriate. 

If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 

Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 

answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

a/ 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  14:  Improve  Organization's  Systems  Engineering 

Processes,  continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments:  _ _ 

ID  Do  those  involved  in  managing  processes  based  on  these  V 

base  practices  ... 

2.1.1  Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2  Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2. 1 .3  Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4  Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5  Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6  Plan  the  performance  of  the  process  area? 

2.4. 1  Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2  Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2  Tailor  the  organization's  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3  Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1  Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2  Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2  Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3  Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  14:  Improve  Organization's  Systems  Engineering 
Processes,  Continued 


4  .  Supporting  Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
infrastructure  analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
„  visible  to  the  organization  as  a  whole. 

Comments:  ^ 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3.1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization’s 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  14:  Improve  Organization's  Systems  Engineering  Processes 
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PA  15:  Manage  Product  Line  Evolution 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  purpose  of  Manage  Product  Line  Evolution  is  to  establish  and  provide  the 
necessary  resources  for  acquiring,  developing,  and  applying  technology  to  a 
product  line  for  competitive  advantage. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 
[]  [] 
[]  [] 

[]  [] 

[]  [] 
[]  [] 


N/A 

[  ]  Define  the  types  of  products  to  be  offered. 

[  ]  Identify  new  product  technologies  that  will  help  the  organization  acquire,  develop, 
and  apply  technology  for  competitive  advantage. 

[  ]  Make  the  necessary  changes  in  the  product  development  cycle  to  support  the 
development  of  new  products. 

[  ]  Ensure  critical  components  are  available  to  support  planned  product  evolution. 

[  ]  Manage  the  insertion  of  new  technology  into  product  development,  marketing,  and 
manufacturing  processes. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes”.  Feel  free  to  add  clarifying  comments  in  the  margin. 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 


SECMM-94-06ICMU/SEI-94-HB-05  v1.0 


A-137 


PA  15:  Manage  Product  Line  Evolution  ,  Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization’s  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  15:  Manage  Product  Line  Evolution,  continued 


4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

V 

3,1.1 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  15:  Manage  Product  Evolution 
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PA  16:  Manage  Systems  Engineering  Support  Environment 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


The  purpose  of  Manage  Systems  Engineering  Support  Environment  is  to  provide 
the  technology  environment  needed  to  develop  the  product  and  perform  the  process. 
The  insertion  of  development  and  process  technology  into  the  environment  is 
executed  with  a  goal  of  minimizing  disruption  of  development  activities  while 
upgrading  to  make  new  technology  available. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
—  it*s  enough  that  it  is  known  who  performs  it. 


Yes  No 

[]  [] 

[]  [] 

[]  [] 

[]  [] 

[]  [] 
[]  [] 

[]  [] 

[]  [] 


N/A 

[  ]  Maintain  awareness  of  the  technologies  that  support  the  organization's  business 
goals. 

[  ]  Determine  requirements  for  the  organization's  systems  engineering  support 
environment  based  on  organizational  needs. 

[  ]  Assess  the  systems  engineering  support  environment  against  the  support 
environment  requirements. 

[  ]  Obtain  a  systems  engineering  support  environment  that  meets  the  requirements  for 
supporting  systems  engineering  by  using  the  practices  in  the  Analyze  Candidate 
Solutions  process  area. 

[  ]  Tailor  the  systems  engineering  support  environment  to  individual  project  needs. 

[  ]  Insert  new  technologies  into  the  systems  engineering  support  environment  based  on 
the  organization's  business  goals  and  the  projects'  needs. 

[  ]  Maintain  the  systems  engineering  support  environment  to  continuously  support  the 
projects  dependent  on  it. 

[  ]  Monitor  the  systems  engineering  support  environment  for  improvement 
opportunities. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


continued  on  next  page 
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PA  16:  Manage  Systems  Engineering  Support  Environment, 

Continued 


2 .  Performing  Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 

the  work  performed  as  part  of  doing  the  work  associated  with  this  process  area. 

Comments: _ _ _ _ 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3.2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  16:  Manage  Systems  Engineering  Support  Environment, 

Continued 


3  .  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

V 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization’s  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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PA  16:  Manage  Systems  Engineering  Support  Environment, 

Continued 


4  .  Supporting  Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
infrastructure  analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
^  visible  to  the  organization  as  a  whole. 

Comments:  ^ 

ID  Are  the  following  visible  and  available  to  those  using  the  V 

organization's  processes? 

3.1.1  Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1  Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1  Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 


end  of  PA  16:  Manage  Systems  Engineering  Support  Environment 
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PA  17:  Manage  Systems  Engineering  Training 


Process  area 
summary 


1 .  Base 
practices 

Comments: 


2 .  Performing 
the  work 

Comments: 


The  purpose  of  Manage  Systems  Engineering  Training  is  to  ensure  that  individuals 
within  the  organization  have  the  necessary  skill  mix  to  perform  their  assigned  tasks 
effectively.  To  achieve  this  objective,  the  skill  requirements  for  the  systems 
engineering  and  related  positions  within  the  organization  need  to  be  identified,  as 
well  as  the  specific  project's  or  organization's  needs  such  as  emergent  technology 
and  new  products,  processes,  and  policies. 


Are  the  practices  identified  below  performed  as  part  of  your  project?  Please  note: 
you  do  not  have  to  personally  be  involved  in  performing  the  practice 
”  it's  enough  that  it  is  known  who  performs  it. 


Yes  No 

[]  [] 

[]  [] 
[]  [] 

[]  [] 
[]  [] 
[]  11 


N/A 

[  ]  Identify  training  needs  throughout  the  organization  using  the  projects’  needs, 
organizational  strategic  plan,  and  existing  employee  skills  as  guidance. 

[  ]  Prepare  training  materials  based  upon  the  identified  training  needs. 

[  ]  Train  personnel  to  have  the  skills  and  knowledge  needed  to  perform  their  assigned 
roles. 

[  ]  Assess  the  effectiveness  of  the  training  to  meet  the  identified  training  needs. 

[  ]  Maintain  records  of  training  and  experience. 

[  ]  Maintain  training  materials  in  an  accessible  repository. 


If  all  of  your  answers  were  "No"  or  "N/A",  skip  to  the  next  process  area. 
Otherwise,  answer  the  following  questions  in  relation  to  the  practices  for  which  you 
answered  "Yes".  Feel  free  to  add  clarifying  comments  in  the  margin. 


Indicate  (V,  or  ?  if  not  known)  if  the  following  SE-CMM  generic  practices  are 
performed  as  part  of  doing  the  work  associated  with  this  process  area. 


ID 

Do  those  involved  in  performing  the  base  practices  of 
this  process  area  ... 

V 

2.2.1 

Use  documented  plans,  standards,  and/or  procedures  in  implementing  the  process 
area? 

2.2.2 

Place  work  products  of  the  process  area  under  version  control  or  configuration 
management,  as  appropriate? 

2.3.1 

Verify  compliance  of  the  process  with  applicable  standards  and/or  procedures? 

2.3,2 

Verify  compliance  of  work  products  with  the  applicable  standards  and/or 
requirements? 

3.2.1 

Use  a  well-defined  process  in  implementing  the  process  area? 

3.2.2 

Perform  defect  reviews  of  appropriate  work  products  of  the  process  area? 

5.2.1 

Perform  causal  analysis  of  defects? 

continued  on  next  page 
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PA  17:  Manage  Systems  Engineering  Training,  Continued 


3.  Process  Indicate  (V,  or  ?  if  not  known)  if  the  process  associated  with  the  Analyze  Candidate 

management  Solutions  process  area  is  managed  using  the  following  generic  practices. 

Comments: 


ID 

Do  those  involved  in  managing  processes  based  on  these 

base  practices  ... 

2.1.1 

Allocate  adequate  resources  (including  people)  for  performing  the  process  area? 

2.1.2 

Assign  responsibilities  for  developing  the  work  products  and/or  providing  the 
services  of  the  process  area? 

2.1.3 

Document  the  approach  to  performing  the  process  area  in  standards  and/or 
procedures? 

2.1.4 

Provide  appropriate  tools  to  support  performance  of  the  process  area? 

2.1.5 

Ensure  that  the  individuals  performing  the  process  area  are  appropriately  trained  in 
how  to  perform  the  process? 

2.1.6 

Plan  the  performance  of  the  process  area? 

2.4.1 

Track  the  status  of  the  process  area  against  the  plan  using  measurement? 

2.4.2 

Take  corrective  action  as  appropriate  when  progress  varies  significantly  from  that 
planned? 

3.1.2 

Tailor  the  organization’s  standard  process  family  to  create  a  defined  process  that 
addresses  the  particular  needs  of  a  specific  use? 

3.2.3 

Use  data  on  performing  the  defined  process  to  manage  it? 

4.2.1 

Determine  the  process  capability  of  the  defined  process  quantitatively? 

4.2.2 

Take  corrective  action  as  appropriate  when  the  process  is  not  performing  within 
its  process  capability? 

5.2.2 

Eliminate  the  causes  of  defects  in  the  defined  process  selectively? 

5.2.3 

Continuously  improve  process  performance  by  changing  the  defined  process  to 
increase  its  effectiveness? 

continued  on  next  page 
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4 .  Supporting 
infrastructure 

Comments: 


Indicate  (V,  or  ?  if  not  known)  if  the  organizational  infrastructure  supports 
analyzing  candidate  solutions  by  making  the  activities  related  to  these  processes 
visible  to  the  organization  as  a  whole. 


ID 

Are  the  following  visible  and  available  to  those  using  the 
organization's  processes? 

3. LI 

Documenting  a  standard  process  or  family  of  processes  for  the  organization,  that 
describes  how  to  implement  the  base  practices  of  the  process  area? 

4.1.1 

Establishing  measurable  quality  goals  for  the  work  products  of  the  organization's 
standard  process  family? 

5.1.1 

Establishing  quantitative  goals  for  improving  process  effectiveness  of  the  standard 
process  family,  based  on  the  business  goals  of  the  organization  and  the  current 
process  capability? 

end  of  PA  17:  Manage  Systems  Engineering  Training 
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